feat: agent self-update across all pull modes — K8s (Helm) + OS-service launcher (Linux/macOS/Windows)#69
Open
yonatan-alien wants to merge 52 commits into
Conversation
added 7 commits
June 2, 2026 15:43
…IEN-59)
Squashed replay of the K8s self-update feature onto current main:
- `alien-agent`: new `agent_upgrade` loop that consumes `agent_target.helm`
from /v1/sync and spawns a Helm-upgrade Job using the upgrader SA; sync
loop persists deployment-scoped token across restarts.
- `alien-helm`: chart guardrails — `required management.{url,name,token,
deploymentId}`, /livez + /readyz probes, Recreate strategy, upgrader SA
+ namespace-scoped Role, `upgrader.enabled` values block.
- `alien-core`: sync schema gains `agent_version`, `agent_os`, `agent_arch`,
`regime`, `agent_image_repository`; K8s worker transport set to `local`.
- `alien-manager`: PUT /v1/deployments/:id/target-agent-version; sync emits
`agent_target` when `target_agent_version` differs from the agent's
reported version; sqlite store + migrations + ReconcileData carry the
new inventory; sync-token persistence reuses across restarts.
- `alien-infra`: K8s ServiceAccount controller is skipped — the chart owns
it, controllers reference it by name.
- `alien-deployment`: Kubernetes platform routes through local env-info
collection (k8s-on-cloud picks the base cloud via existing context).
Copy the regenerated `apps/api/openapi.json` from the platform repo so the Rust SDK (`alien-platform-api`, generated via `progenitor` at build time) picks up the new `PUT /v1/deployments/:id/target-agent-version` endpoint, the extended SyncReconcileRequest fields, and the `basePlatform` property on `PersistImportedDeploymentRequest`.
Without a PVC backing `/var/lib/alien-agent`, the agent's `deployment_id` + deployment-scoped sync token live in an `emptyDir` and are wiped on any pod restart. The self-update flow (manager emits `agent_target.helm` → agent spawns helm-upgrade Job → Job rolls the agent Deployment) triggers exactly such a restart, and the new pod re-runs `/v1/initialize`, hits a DEPLOYMENT_NAME_ALREADY_EXISTS 409, CrashLoopBackOffs, helm `--atomic` times out, and rolls back. The feature is silently unusable. Default `runtime.data.persistence.enabled` to `true` so self-update survives the pod roll on any cluster with a default StorageClass. Operators on clusters without one can still set `storageClassName`, point at an `existingClaim`, or explicitly opt out and accept that self-update won't survive a pod roll. Helm snapshot tests rebaselined.
The agent's `data_dir` (`/var/lib/alien-agent`) holds its persistent
`deployment_id` + deployment-scoped sync token. When the chart leaves
`runtime.data.persistence` off (or anything else wipes that directory —
pod eviction, manual reset, etc.) the agent restarts with no stored
state and falls into the "first startup" branch, hits `/v1/initialize`,
and the platform rejects with `DEPLOYMENT_NAME_ALREADY_EXISTS 409`
because the row is still there. Result: CrashLoopBackOff, and if the
restart was driven by self-update's helm-upgrade Job, helm `--atomic`
times out and rolls back.
Add `POST /v1/rejoin` so the agent can explicitly re-attach to an
existing deployment by name. Same dg bearer the chart mounts, no body
beyond the name; response is `{ deploymentId, token }` with a freshly
minted deployment-scoped token. The agent code-path catches the 409
from initialize and falls through to rejoin instead of erroring.
- `alien-manager`: new `RejoinRequest`/`RejoinResponse`, `rejoin`
handler that mirrors the existing initialize idempotency branch
(look up by `(dg, name)`, mint a fresh token), wired through a new
`rejoin_router()` so multi-tenant embedders can override it the same
way they override initialize. `RouterOptions::include_rejoin` + a
`skip_rejoin()` builder method follow the existing pattern.
- `alien-agent`: on 409 from initialize, resolve the deployment name
the same way initialize did (CLI arg / env / hostname) and call the
new `rejoin_with_manager` helper. The recovered `(deployment_id,
token)` are written back to `data_dir` like a normal init, so
subsequent restarts go through the stored-state branch.
- `alien-manager-api` SDK regenerated; OSS `/v1/rejoin` is exposed in
the openapi schema for downstream tooling.
The agent's initialize → rejoin fall-through keyed off `http_status_code == Some(409)`, but `initialize_with_manager` wrapped every SDK error in `ConfigurationError` whose hardcoded `http_status_code = 500` clobbered the real status. Result: agent received "Unexpected response: 409 Conflict" from the manager, hit the `.context(ConfigurationError)` wrap, the outer status became 500, and the agent crashed instead of falling through to `/v1/rejoin`. - New `DeploymentNameAlreadyExists` variant on `ErrorData` (409, non-retryable, non-internal) so the 409 path has a distinct outer error. - `initialize_with_manager` rebuilt: 409 from the SDK is re-emitted as `DeploymentNameAlreadyExists`; everything else still goes through `ConfigurationError` so existing error paths are unchanged. Confirmed live: after wiping `/var/lib/alien-agent/*` and bouncing the pod, the agent now logs `Name already exists — assuming local state was wiped, rejoining…`, calls `/v1/rejoin`, recovers the same deployment_id, and stays running.
…-update-for-the-alien-agent-across-all-pull-mode-2 # Conflicts: # client-sdks/platform/openapi.json # client-sdks/platform/rust/openapi-3.0.json # client-sdks/platform/rust/openapi.json # crates/alien-helm/src/generator.rs # crates/alien-helm/tests/generator/boot_paths_tests.rs # crates/alien-helm/tests/generator/snapshots/generator__generator__helpers__data_layer.snap # crates/alien-helm/tests/generator/snapshots/generator__generator__helpers__manager_only_pure_worker.snap
Greptile SummaryThis PR lands manager-driven K8s agent self-update (Helm-runner Job spawned from the agent on
Confidence Score: 3/5The self-update happy path is solid, but a failed helm-runner Job silently blocks all retry attempts for up to 10 minutes with no operator warning. The core upgrade loop, rejoin recovery, chart guardrails, and persistence defaults are well-reasoned and tested. The 409-on-failed-Job case in create_job quietly absorbs what should be a retryable error, and min_supported_version being pinned to the target version would block all upgrades once the agent enforces that field. The upgrade path is a new critical flow touching the agent self-replacement mechanism, so these gaps carry more weight than they would in a peripheral feature. crates/alien-agent/src/loops/agent_upgrade.rs (failed-Job 409 handling) and crates/alien-manager/src/routes/sync.rs (build_agent_target min_supported_version) need the most attention before this ships to production.
|
| Filename | Overview |
|---|---|
| crates/alien-agent/src/loops/agent_upgrade.rs | New Helm-runner Job spawner; 409 from a failed (but not yet TTL'd) Job is silently treated as idempotent success, stalling upgrades for up to 10 min with no operator warning. |
| crates/alien-manager/src/routes/sync.rs | New agent_sync inventory persistence and build_agent_target; min_supported_version is set equal to the target version, which would block all agents from upgrading once that field is enforced. |
| crates/alien-manager/src/routes/deployments.rs | New PUT /v1/deployments/:id/target-agent-version endpoint; auth, authz, and semver validation are present; is_plausible_semver rejects valid versions with both pre-release and build metadata. |
| crates/alien-helm/src/generator.rs | Chart guardrails (required fields, Recreate strategy, persistence default, upgrader SA/Role/ClusterRole), encryption key auto-generation, and /livez+/readyz probe defaults all look correct. |
| crates/alien-agent/src/cli.rs | Adds rejoin fall-through on 409 from initialize and sync-token persistence; logic and error routing look correct. |
| crates/alien-manager/src/stores/sqlite/deployment.rs | update_agent_metadata and set_target_agent_version are correctly scoped; neither updates the updated_at column, leaving deployment timestamps stale after inventory syncs. |
| crates/alien-core/src/sync.rs | SyncRequest/SyncResponse extended with new optional fields; AgentTarget, AgentHelmTarget, AgentBinaryTarget structs and back-compat tests look correct. |
| crates/alien-infra/src/core/executor.rs | Kubernetes ServiceAccounts are correctly skipped at controller-lookup time and auto-marked Running as platform-provided resources; logic mirrors external-binding treatment. |
Sequence Diagram
sequenceDiagram
participant Agent
participant Manager
participant K8sAPI
participant HelmRunnerJob
Note over Agent,Manager: Normal sync loop
Agent->>Manager: "POST /v1/sync (agent_version, regime=kubernetes)"
Manager->>Manager: update_agent_metadata()
Manager->>Manager: build_agent_target() if target differs from reported
Manager-->>Agent: SyncResponse with agent_target helm payload
Note over Agent,K8sAPI: Self-update actuator
Agent->>K8sAPI: POST /apis/batch/v1/.../jobs (helm-runner Job)
K8sAPI-->>Agent: 201 Created (or 409 if exists)
K8sAPI->>HelmRunnerJob: schedule pod
HelmRunnerJob->>HelmRunnerJob: helm upgrade --reuse-values --atomic --wait
Note over Agent,Manager: Pod rolls (Recreate strategy)
HelmRunnerJob->>K8sAPI: Deployment updated with new image tag
Agent->>Agent: /readyz returns 503 until first sync
Agent->>Manager: POST /v1/sync with new agent_version
Agent->>Agent: "first_sync_completed=true, /readyz returns 200"
Note over Agent,Manager: Rejoin after state wipe
Agent->>Manager: POST /v1/initialize returns 409
Agent->>Manager: POST /v1/rejoin with name
Manager->>Manager: get_deployment_by_name() and mint new token
Manager-->>Agent: deployment_id and fresh token
Agent->>Agent: db.set_sync_token() persists for next restart
Comments Outside Diff (1)
-
crates/alien-manager/src/stores/sqlite/deployment.rs, line 506-550 (link)updated_atis not bumped during agent metadata writesupdate_agent_metadata(and similarlyset_target_agent_version) never touches theupdated_atcolumn. Any API consumer that usesupdated_atto detect "something changed on this deployment" will miss both inventory syncs and operator target-version pins. Consider settingupdated_at = datetime('now')inside the same UPDATE, or at least doing so forset_target_agent_versionwhich represents an explicit admin action that callers would expect to be reflected in the record's timestamp.Prompt To Fix With AI
This is a comment left during a code review. Path: crates/alien-manager/src/stores/sqlite/deployment.rs Line: 506-550 Comment: **`updated_at` is not bumped during agent metadata writes** `update_agent_metadata` (and similarly `set_target_agent_version`) never touches the `updated_at` column. Any API consumer that uses `updated_at` to detect "something changed on this deployment" will miss both inventory syncs and operator target-version pins. Consider setting `updated_at = datetime('now')` inside the same UPDATE, or at least doing so for `set_target_agent_version` which represents an explicit admin action that callers would expect to be reflected in the record's timestamp. How can I resolve this? If you propose a fix, please make it concise.
Prompt To Fix All With AI
Fix the following 4 code review issues. Work through them one at a time, proposing concise fixes.
---
### Issue 1 of 4
crates/alien-agent/src/loops/agent_upgrade.rs:296-305
**Silently swallows a failed Job's 409 — upgrade stalls with no warning**
When the Helm-runner Job exists but is in a `Failed` state (exhausted its `backoffLimit`), the Kubernetes API still returns 409 on a new create. The code logs at `info` level and returns `Ok(())`, so `apply_agent_target` sees success. The manager keeps emitting `agent_target` on every sync tick, but the agent will keep hitting the same 409 and silently doing nothing for up to `ttlSecondsAfterFinished` seconds (10 minutes). During that window, there is no `warn` log and no error propagated — an operator watching logs will only see "Upgrader Job already exists (409); leaving in place" with no indication the actual upgrade never happened.
To distinguish "another sync already queued it" (desired idempotency) from "it exists but is already dead," the handler should query the Job status and emit at least a `warn` when the existing Job is in a `Failed` condition, so operators can investigate before the TTL window expires.
### Issue 2 of 4
crates/alien-manager/src/routes/sync.rs:1107-1112
`min_supported_version` is set equal to `version`, which will break upgrades once the field is enforced. The field is described as "the agent should refuse the upgrade if its own version is older than this." Setting it to the target version means only an agent already at version X would accept the upgrade to X — but those agents never receive the instruction (the `reported_version == Some(target_version)` guard returns `None` first). Every agent that actually needs upgrading would refuse. The MVP intent (allow all current agents to upgrade) is better expressed as an empty floor like `"1.0.0"`, or by explicitly recording the minimum agent version that is safe to self-upgrade.
```suggestion
Some(alien_core::sync::AgentTarget {
version: target_version.to_string(),
// Allow all existing agents to receive the upgrade instruction.
// Set to a meaningful floor (e.g. the first version that supports
// self-update) once the agent enforces this field.
min_supported_version: "1.0.0".to_string(),
binary: None,
helm,
})
```
### Issue 3 of 4
crates/alien-manager/src/routes/deployments.rs:876-892
`splitn(2, |c| c == '-' || c == '+')` splits on the **first** occurrence of either separator, so versions with both a pre-release suffix and build metadata (e.g. `1.4.0-rc.1+build.123`) fail: the remainder becomes `"rc.1+build.123"` and the `+` isn't in the allowed character set `[alphanumeric, '.', '-']`. The function correctly rejects garbage input, but it silently blocks valid SemVer 2.0.0 strings that operators may reasonably try to pin.
```suggestion
fn is_plausible_semver(v: &str) -> bool {
// Strip optional +build metadata first so the pre-release check can
// use a simple split on '-'.
let without_build = v.splitn(2, '+').next().unwrap_or(v);
let mut parts = without_build.splitn(2, '-');
let core = parts.next().unwrap_or("");
let core_ok = {
let segs: Vec<&str> = core.split('.').collect();
segs.len() == 3
&& segs
.iter()
.all(|s| !s.is_empty() && s.chars().all(|c| c.is_ascii_digit()))
};
let rest_ok = parts.next().map_or(true, |r| {
!r.is_empty()
&& r.chars()
.all(|c| c.is_ascii_alphanumeric() || c == '.' || c == '-')
});
core_ok && rest_ok
}
```
### Issue 4 of 4
crates/alien-manager/src/stores/sqlite/deployment.rs:506-550
**`updated_at` is not bumped during agent metadata writes**
`update_agent_metadata` (and similarly `set_target_agent_version`) never touches the `updated_at` column. Any API consumer that uses `updated_at` to detect "something changed on this deployment" will miss both inventory syncs and operator target-version pins. Consider setting `updated_at = datetime('now')` inside the same UPDATE, or at least doing so for `set_target_agent_version` which represents an explicit admin action that callers would expect to be reflected in the record's timestamp.
Reviews (1): Last reviewed commit: "Merge remote-tracking branch 'origin/mai..." | Re-trigger Greptile
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
Comment on lines
+296
to
+305
| } | ||
| out | ||
| } | ||
|
|
||
| #[cfg(test)] | ||
| mod tests { | ||
| use super::*; | ||
| use serde_json::json; | ||
|
|
||
| fn inputs() -> JobInputs { |
There was a problem hiding this comment.
When the Helm-runner Job exists but is in a Failed state (exhausted its backoffLimit), the Kubernetes API still returns 409 on a new create. The code logs at info level and returns Ok(()), so apply_agent_target sees success. The manager keeps emitting agent_target on every sync tick, but the agent will keep hitting the same 409 and silently doing nothing for up to ttlSecondsAfterFinished seconds (10 minutes). During that window, there is no warn log and no error propagated — an operator watching logs will only see "Upgrader Job already exists (409); leaving in place" with no indication the actual upgrade never happened.
To distinguish "another sync already queued it" (desired idempotency) from "it exists but is already dead," the handler should query the Job status and emit at least a warn when the existing Job is in a Failed condition, so operators can investigate before the TTL window expires.
Prompt To Fix With AI
This is a comment left during a code review.
Path: crates/alien-agent/src/loops/agent_upgrade.rs
Line: 296-305
Comment:
**Silently swallows a failed Job's 409 — upgrade stalls with no warning**
When the Helm-runner Job exists but is in a `Failed` state (exhausted its `backoffLimit`), the Kubernetes API still returns 409 on a new create. The code logs at `info` level and returns `Ok(())`, so `apply_agent_target` sees success. The manager keeps emitting `agent_target` on every sync tick, but the agent will keep hitting the same 409 and silently doing nothing for up to `ttlSecondsAfterFinished` seconds (10 minutes). During that window, there is no `warn` log and no error propagated — an operator watching logs will only see "Upgrader Job already exists (409); leaving in place" with no indication the actual upgrade never happened.
To distinguish "another sync already queued it" (desired idempotency) from "it exists but is already dead," the handler should query the Job status and emit at least a `warn` when the existing Job is in a `Failed` condition, so operators can investigate before the TTL window expires.
How can I resolve this? If you propose a fix, please make it concise.
Comment on lines
+1107
to
+1112
| Some(alien_core::sync::AgentTarget { | ||
| version: target_version.to_string(), | ||
| min_supported_version: target_version.to_string(), | ||
| binary: None, | ||
| helm, | ||
| }) |
There was a problem hiding this comment.
min_supported_version is set equal to version, which will break upgrades once the field is enforced. The field is described as "the agent should refuse the upgrade if its own version is older than this." Setting it to the target version means only an agent already at version X would accept the upgrade to X — but those agents never receive the instruction (the reported_version == Some(target_version) guard returns None first). Every agent that actually needs upgrading would refuse. The MVP intent (allow all current agents to upgrade) is better expressed as an empty floor like "1.0.0", or by explicitly recording the minimum agent version that is safe to self-upgrade.
Suggested change
| Some(alien_core::sync::AgentTarget { | |
| version: target_version.to_string(), | |
| min_supported_version: target_version.to_string(), | |
| binary: None, | |
| helm, | |
| }) | |
| Some(alien_core::sync::AgentTarget { | |
| version: target_version.to_string(), | |
| // Allow all existing agents to receive the upgrade instruction. | |
| // Set to a meaningful floor (e.g. the first version that supports | |
| // self-update) once the agent enforces this field. | |
| min_supported_version: "1.0.0".to_string(), | |
| binary: None, | |
| helm, | |
| }) |
Prompt To Fix With AI
This is a comment left during a code review.
Path: crates/alien-manager/src/routes/sync.rs
Line: 1107-1112
Comment:
`min_supported_version` is set equal to `version`, which will break upgrades once the field is enforced. The field is described as "the agent should refuse the upgrade if its own version is older than this." Setting it to the target version means only an agent already at version X would accept the upgrade to X — but those agents never receive the instruction (the `reported_version == Some(target_version)` guard returns `None` first). Every agent that actually needs upgrading would refuse. The MVP intent (allow all current agents to upgrade) is better expressed as an empty floor like `"1.0.0"`, or by explicitly recording the minimum agent version that is safe to self-upgrade.
```suggestion
Some(alien_core::sync::AgentTarget {
version: target_version.to_string(),
// Allow all existing agents to receive the upgrade instruction.
// Set to a meaningful floor (e.g. the first version that supports
// self-update) once the agent enforces this field.
min_supported_version: "1.0.0".to_string(),
binary: None,
helm,
})
```
How can I resolve this? If you propose a fix, please make it concise.
Comment on lines
+876
to
+892
| fn is_plausible_semver(v: &str) -> bool { | ||
| let mut parts = v.splitn(2, |c| c == '-' || c == '+'); | ||
| let core = parts.next().unwrap_or(""); | ||
| let core_ok = { | ||
| let segs: Vec<&str> = core.split('.').collect(); | ||
| segs.len() == 3 | ||
| && segs | ||
| .iter() | ||
| .all(|s| !s.is_empty() && s.chars().all(|c| c.is_ascii_digit())) | ||
| }; | ||
| let rest_ok = parts.next().map_or(true, |r| { | ||
| !r.is_empty() | ||
| && r.chars() | ||
| .all(|c| c.is_ascii_alphanumeric() || c == '.' || c == '-') | ||
| }); | ||
| core_ok && rest_ok | ||
| } |
There was a problem hiding this comment.
splitn(2, |c| c == '-' || c == '+') splits on the first occurrence of either separator, so versions with both a pre-release suffix and build metadata (e.g. 1.4.0-rc.1+build.123) fail: the remainder becomes "rc.1+build.123" and the + isn't in the allowed character set [alphanumeric, '.', '-']. The function correctly rejects garbage input, but it silently blocks valid SemVer 2.0.0 strings that operators may reasonably try to pin.
Suggested change
| fn is_plausible_semver(v: &str) -> bool { | |
| let mut parts = v.splitn(2, |c| c == '-' || c == '+'); | |
| let core = parts.next().unwrap_or(""); | |
| let core_ok = { | |
| let segs: Vec<&str> = core.split('.').collect(); | |
| segs.len() == 3 | |
| && segs | |
| .iter() | |
| .all(|s| !s.is_empty() && s.chars().all(|c| c.is_ascii_digit())) | |
| }; | |
| let rest_ok = parts.next().map_or(true, |r| { | |
| !r.is_empty() | |
| && r.chars() | |
| .all(|c| c.is_ascii_alphanumeric() || c == '.' || c == '-') | |
| }); | |
| core_ok && rest_ok | |
| } | |
| fn is_plausible_semver(v: &str) -> bool { | |
| // Strip optional +build metadata first so the pre-release check can | |
| // use a simple split on '-'. | |
| let without_build = v.splitn(2, '+').next().unwrap_or(v); | |
| let mut parts = without_build.splitn(2, '-'); | |
| let core = parts.next().unwrap_or(""); | |
| let core_ok = { | |
| let segs: Vec<&str> = core.split('.').collect(); | |
| segs.len() == 3 | |
| && segs | |
| .iter() | |
| .all(|s| !s.is_empty() && s.chars().all(|c| c.is_ascii_digit())) | |
| }; | |
| let rest_ok = parts.next().map_or(true, |r| { | |
| !r.is_empty() | |
| && r.chars() | |
| .all(|c| c.is_ascii_alphanumeric() || c == '.' || c == '-') | |
| }); | |
| core_ok && rest_ok | |
| } |
Prompt To Fix With AI
This is a comment left during a code review.
Path: crates/alien-manager/src/routes/deployments.rs
Line: 876-892
Comment:
`splitn(2, |c| c == '-' || c == '+')` splits on the **first** occurrence of either separator, so versions with both a pre-release suffix and build metadata (e.g. `1.4.0-rc.1+build.123`) fail: the remainder becomes `"rc.1+build.123"` and the `+` isn't in the allowed character set `[alphanumeric, '.', '-']`. The function correctly rejects garbage input, but it silently blocks valid SemVer 2.0.0 strings that operators may reasonably try to pin.
```suggestion
fn is_plausible_semver(v: &str) -> bool {
// Strip optional +build metadata first so the pre-release check can
// use a simple split on '-'.
let without_build = v.splitn(2, '+').next().unwrap_or(v);
let mut parts = without_build.splitn(2, '-');
let core = parts.next().unwrap_or("");
let core_ok = {
let segs: Vec<&str> = core.split('.').collect();
segs.len() == 3
&& segs
.iter()
.all(|s| !s.is_empty() && s.chars().all(|c| c.is_ascii_digit()))
};
let rest_ok = parts.next().map_or(true, |r| {
!r.is_empty()
&& r.chars()
.all(|c| c.is_ascii_alphanumeric() || c == '.' || c == '-')
});
core_ok && rest_ok
}
```
How can I resolve this? If you propose a fix, please make it concise.
alongubkin
requested changes
Jun 4, 2026
| /// POST a Job to the in-pod Kubernetes API. Returns Ok on 2xx, AlienError | ||
| /// otherwise. 409 (already exists) is treated as success — another sync | ||
| /// already kicked the Job and idempotency handles the retry. | ||
| async fn create_job(namespace: &str, body: &serde_json::Value) -> Result<()> { |
Member
There was a problem hiding this comment.
these APIs should be in alien-k8s-clients
| pub agent_version: Option<String>, | ||
| /// `linux` / `macos` / `windows`. From `std::env::consts::OS`. | ||
| #[serde(default, skip_serializing_if = "Option::is_none")] | ||
| pub agent_os: Option<String>, |
Member
There was a problem hiding this comment.
I think we have an existing enum for this somewhere in alien-core
Contributor
Author
There was a problem hiding this comment.
There is only Architecture for cloud instances (Arm64/X86_64). I'll make enums for the agent arch/os
| pub agent_os: Option<String>, | ||
| /// `x86_64` / `aarch64`. From `std::env::consts::ARCH`. | ||
| #[serde(default, skip_serializing_if = "Option::is_none")] | ||
| pub agent_arch: Option<String>, |
Member
There was a problem hiding this comment.
I think we have an existing enum for this somewhere in alien-core
Contributor
Author
There was a problem hiding this comment.
There is only Architecture for cloud instances (Arm64/X86_64)
| /// How the agent is supervised — `os-service` (launcher) or `kubernetes` | ||
| /// (Helm). Detected at runtime from `KUBERNETES_SERVICE_HOST`. | ||
| #[serde(default, skip_serializing_if = "Option::is_none")] | ||
| pub regime: Option<AgentRegime>, |
Member
There was a problem hiding this comment.
regime is a really weird name haha
Contributor
Author
There was a problem hiding this comment.
I'll change to packaging
| exec helm upgrade \"$RELEASE\" \"$CHART_REF\"{version_flag}{extra_args_flag} \ | ||
| --namespace \"$NAMESPACE\" \ | ||
| --reuse-values \ | ||
| --atomic --wait \ |
Member
There was a problem hiding this comment.
Can you document why did we choose --atomic?
| --values /tmp/values.json" | ||
| ); | ||
|
|
||
| let body = serde_json::json!({ |
Member
There was a problem hiding this comment.
What happens if creating this job fails? Or pulling the job's image fails? Or the command inside it fails? Do we report back those failures in any way?
How many retries do we support (I think the answer should be unlimited)? Is there exponential backoff or constant interval between retries?
| "name": job_name, | ||
| "namespace": inputs.namespace, | ||
| "labels": { | ||
| "app.kubernetes.io/managed-by": "alien-agent", |
Member
There was a problem hiding this comment.
We recently improved our white-labeling support, need to make sure it fits there. See other places in alien-helm
| @@ -0,0 +1,453 @@ | |||
| //! Agent self-update actuator. When `/v1/sync` returns | |||
Member
There was a problem hiding this comment.
Right now this will only work for Kubernetes deployments, right? No support for local service swap etc? What'll happen in local?
Contributor
Author
There was a problem hiding this comment.
As decided, we'll also include service implementation for Linux/Windows/Mac in this branch before merging.
Conflict resolutions: - cli.rs: keep main's select_startup_deployment_id refactor; re-apply our sync-token restore, stack-settings forwarding, and 409->rejoin fallthrough into its branches. Union imports (PublicEndpointUrls + ContextError). - sync.rs / executor.rs: disjoint additions, kept both. - ReconcileData / DeploymentRecord gained agent-inventory fields on our branch; set them to None at main's reconcile-loop site and two test fixtures (no agent sync there). - manager openapi (crates + client-sdks/manager): regenerated from merged Rust via generate:manager-rust-sdk. - client-sdks/platform/*: took main's (superset); pending re-sync from the platform-side openapi regen. Validated: cargo test -p alien-agent -p alien-core -p alien-deployment -p alien-helm -p alien-manager --lib -> 518 passed, 0 failed. NOTE: manager TypeScript SDK not regenerated (speakeasy needs interactive auth); rerun `pnpm run generate:manager-api` after `speakeasy auth login`. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Replaces the main-only platform spec taken during the origin/main merge with the freshly regenerated apps/api/openapi.json (main + the ALIEN-59 rejoin / target-agent-version / agent-inventory additions). Closes the "next platform regen will bring main's schemas back" note from the merge. Validated: cargo check -p alien-platform-api -p alien-bindings. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Resolve sync.rs (keep detect_agent_regime alongside apply_manager_control_state) and regenerate the platform API specs and Cargo.lock from the merged code. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
yonatan-alien
changed the title
…-update-for-the-alien-agent-across-all-pull-mode-2 # Conflicts: # Cargo.lock # client-sdks/platform/openapi.json # client-sdks/platform/rust/openapi-3.0.json # client-sdks/platform/rust/openapi.json # client-sdks/platform/typescript/.speakeasy/gen.lock # client-sdks/platform/typescript/.speakeasy/gen.yaml # client-sdks/platform/typescript/README.md # client-sdks/platform/typescript/jsr.json # client-sdks/platform/typescript/package.json # client-sdks/platform/typescript/src/funcs/deploymentGroupsCreateFirstPartyDeploymentSession.ts # client-sdks/platform/typescript/src/funcs/deploymentsSetFirstPartyDeploymentInputs.ts # client-sdks/platform/typescript/src/lib/config.ts # client-sdks/platform/typescript/src/models/index.ts # client-sdks/platform/typescript/src/sdk/deployments.ts
The main-merge SDK regeneration bumped client-sdks/platform/typescript eslint/@eslint/js to ^9.26.0 but the lockfile wasn't refreshed, breaking `pnpm install --frozen-lockfile` in CI. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The chart secret.yaml 'required' guardrail (management.token or existingSecret) rejects partial values documents. Three manager_only render tests passed None/partial values and began failing once the guardrail landed. Render them with the manager-fetch overlay like their siblings (aws now matches the existing gcp test). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Agent → manager reporting of self-update progress/failure so the platform can show a truthful failed/in-progress state instead of inferring from a stalled version — implements the 'update_failed' signal the self-update spec already referenced. - alien-core: SyncRequest.agent_update + AgentUpdateReport/AgentUpdatePhase - alien-agent: status-aware upgrader reconcile — unique per-attempt Job names, Job/pod status reads, Pull/Apply classification, exponential backoff, warn on a failed Job, report construction; removes the blind 409=success stall - alien-manager: thread agent_update through AgentSyncRequest + ReconcileData - regenerate manager + platform OpenAPI / Rust + TS SDKs Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…-update-for-the-alien-agent-across-all-pull-mode-2 # Conflicts: # client-sdks/manager/openapi.json # client-sdks/manager/rust/openapi-3.0.json # client-sdks/platform/openapi.json # client-sdks/platform/rust/openapi-3.0.json # client-sdks/platform/rust/openapi.json # client-sdks/platform/typescript/.speakeasy/gen.lock # client-sdks/platform/typescript/.speakeasy/gen.yaml # client-sdks/platform/typescript/docs/models/deployment.md # client-sdks/platform/typescript/docs/models/deploymentdetailresponse.md # client-sdks/platform/typescript/docs/models/deploymentlistitemresponse.md # client-sdks/platform/typescript/docs/models/syncacquireresponseclusterendpointunion.md # client-sdks/platform/typescript/docs/models/syncacquireresponseconfig.md # client-sdks/platform/typescript/docs/models/syncacquireresponsedatabaseunion1.md # client-sdks/platform/typescript/docs/models/syncacquireresponsedatabaseunion2.md # client-sdks/platform/typescript/docs/models/syncacquireresponsedatabaseunion3.md # client-sdks/platform/typescript/docs/models/syncacquireresponsedatabaseunion4.md # client-sdks/platform/typescript/docs/models/syncacquireresponsedatabaseunion5.md # client-sdks/platform/typescript/docs/models/syncacquireresponseexternalbindingsunion7.md # client-sdks/platform/typescript/docs/models/syncacquireresponsehostunion1.md # client-sdks/platform/typescript/docs/models/syncacquireresponsehostunion2.md # client-sdks/platform/typescript/docs/models/syncacquireresponsehostunion3.md # client-sdks/platform/typescript/docs/models/syncacquireresponsehostunion4.md # client-sdks/platform/typescript/docs/models/syncacquireresponsepasswordsecretarnunion.md # client-sdks/platform/typescript/docs/models/syncacquireresponsepasswordsecretnameunion.md # client-sdks/platform/typescript/docs/models/syncacquireresponsepasswordsecreturiunion.md # client-sdks/platform/typescript/docs/models/syncacquireresponseportunion1.md # client-sdks/platform/typescript/docs/models/syncacquireresponseportunion2.md # client-sdks/platform/typescript/docs/models/syncacquireresponseportunion3.md # client-sdks/platform/typescript/docs/models/syncacquireresponseportunion4.md # client-sdks/platform/typescript/docs/models/syncacquireresponseportunion5.md # client-sdks/platform/typescript/docs/models/syncacquireresponseusernameunion1.md # client-sdks/platform/typescript/docs/models/syncacquireresponseusernameunion2.md # client-sdks/platform/typescript/docs/models/syncacquireresponseusernameunion3.md # client-sdks/platform/typescript/docs/models/syncacquireresponseusernameunion4.md # client-sdks/platform/typescript/docs/models/syncacquireresponseusernameunion5.md # client-sdks/platform/typescript/docs/models/synclistresponsedeployment.md # client-sdks/platform/typescript/docs/models/syncreconcilerequest.md # client-sdks/platform/typescript/docs/models/syncreconcileresponseclusterendpointunion.md # client-sdks/platform/typescript/docs/models/syncreconcileresponsedatabaseunion1.md # client-sdks/platform/typescript/docs/models/syncreconcileresponsedatabaseunion2.md # client-sdks/platform/typescript/docs/models/syncreconcileresponsedatabaseunion3.md # client-sdks/platform/typescript/docs/models/syncreconcileresponsedatabaseunion4.md # client-sdks/platform/typescript/docs/models/syncreconcileresponsedatabaseunion5.md # client-sdks/platform/typescript/docs/models/syncreconcileresponseexternalbindingsunion7.md # client-sdks/platform/typescript/docs/models/syncreconcileresponsehostunion1.md # client-sdks/platform/typescript/docs/models/syncreconcileresponsehostunion2.md # client-sdks/platform/typescript/docs/models/syncreconcileresponsehostunion3.md # client-sdks/platform/typescript/docs/models/syncreconcileresponsehostunion4.md # client-sdks/platform/typescript/docs/models/syncreconcileresponsepasswordsecretarnunion.md # client-sdks/platform/typescript/docs/models/syncreconcileresponsepasswordsecretnameunion.md # client-sdks/platform/typescript/docs/models/syncreconcileresponsepasswordsecreturiunion.md # client-sdks/platform/typescript/docs/models/syncreconcileresponseportunion1.md # client-sdks/platform/typescript/docs/models/syncreconcileresponseportunion2.md # client-sdks/platform/typescript/docs/models/syncreconcileresponseportunion3.md # client-sdks/platform/typescript/docs/models/syncreconcileresponseportunion4.md # client-sdks/platform/typescript/docs/models/syncreconcileresponseportunion5.md # client-sdks/platform/typescript/docs/models/syncreconcileresponseusernameunion1.md # client-sdks/platform/typescript/docs/models/syncreconcileresponseusernameunion2.md # client-sdks/platform/typescript/docs/models/syncreconcileresponseusernameunion3.md # client-sdks/platform/typescript/docs/models/syncreconcileresponseusernameunion4.md # client-sdks/platform/typescript/docs/models/syncreconcileresponseusernameunion5.md # client-sdks/platform/typescript/docs/models/targetconfig.md # client-sdks/platform/typescript/jsr.json # client-sdks/platform/typescript/package.json # client-sdks/platform/typescript/src/lib/config.ts # client-sdks/platform/typescript/src/models/createmanagerresponse.ts # client-sdks/platform/typescript/src/models/deployment.ts # client-sdks/platform/typescript/src/models/deploymentdetailresponse.ts # client-sdks/platform/typescript/src/models/managerretryresponse.ts # client-sdks/platform/typescript/src/models/operations/getresourcedeploymentdetail.ts # client-sdks/platform/typescript/src/models/persistimporteddeploymentrequest.ts # client-sdks/platform/typescript/src/models/syncacquireresponse.ts # client-sdks/platform/typescript/src/models/synclistresponse.ts # client-sdks/platform/typescript/src/models/syncreconcilerequest.ts # client-sdks/platform/typescript/src/models/syncreconcileresponse.ts # crates/alien-core/src/sync.rs # crates/alien-helm/src/generator.rs # crates/alien-helm/tests/generator/snapshots/generator__generator__helpers__data_layer.snap # crates/alien-helm/tests/generator/snapshots/generator__generator__helpers__manager_only_pure_worker.snap # crates/alien-manager/openapi.json # crates/alien-manager/src/loops/deployment.rs # crates/alien-manager/src/routes/sync.rs # crates/alien-manager/src/traits/deployment_store.rs # crates/alien-manager/src/transports/manager.rs # crates/alien-manager/tests/registry_proxy_cloud_test.rs # crates/alien-manager/tests/registry_proxy_test.rs # crates/alien-manager/tests/sqlite_store_tests.rs # crates/alien-operator/src/cli.rs # crates/alien-operator/src/db.rs # crates/alien-operator/src/lib.rs # crates/alien-operator/src/loops/agent_upgrade.rs # crates/alien-operator/src/loops/sync.rs # crates/alien-operator/src/otlp_server.rs
…-update-for-the-alien-agent-across-all-pull-mode-2
The platform API renamed agent→operator (self-update fields, target version endpoint); regenerate the distributed platform SDK openapi so the committed client matches. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
A stale earlier lockfile-sync commit pinned client-sdks/platform/typescript eslint specifiers to ^9.26.0, but the package.json is back to main's ^9.19.0, breaking `pnpm install --frozen-lockfile` in CI Fast. No package.json differs from origin/main, so main's lockfile is exactly correct here. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
An earlier speakeasy regen committed a bloated TS SDK (+156k lines of exploded numbered models, e.g. statuslifecycle18.ts), which OOMs tsc even at 8GB heap in CI Fast. Nothing on this branch imports this SDK for compile (managerx uses the Rust SDK; the dashboard has its own client), so restore main's clean SDK + its openapi.json. The Rust SDK openapi keeps the operator content managerx needs; a proper operator TS SDK regen is a release-pipeline follow-up. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
`cargo check` skips #[cfg(test)] modules, so a few test constructions kept the old `regime` field after the rename: SyncRequest field access in alien-core sync tests, and DeploymentRecord `regime: None` inits in three alien-manager test helpers. cargo nextest compiles these, breaking CI Fast's non-cloud rust tests. Verified all non-cloud test targets compile. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Conform the OSS standalone manager to the packaging rename: the sqlite
`regime` column + sea_query `Regime` variant, the update-inventory params,
build_operator_target arg, a test fn name, and comments across manager/
operator/core.
Also fixes a latent bug the operator reconciliation left: the migrations
renamed the inventory columns to operator_* but the sea_query enum still
had Agent* variants (→ "agent_version"), so the inventory queries failed
at runtime ("no such column: agent_version"). Renamed the enum variants to
Operator* to match the columns. This is what broke CI Fast's non-cloud rust
tests (4 sqlite store tests). All 33 sqlite store tests pass.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Mostly clean auto-merge. Manual resolutions: - Platform SDK openapi (client-sdks/platform/rust/*): took main's newer version; regenerated with operator content during the platform merge. - main added a new Platform::Machines enum variant — added the missing match arms in alien-test (config/distribution/e2e), grouping Machines with Kubernetes/Local/Test per the rest of the codebase. - Fixed a latent bug the merge's readyz test caught: the otlp_server test helper ignored its first_sync_completed param and hardcoded `true`, so the /readyz readiness-gate test always saw ready. Pass the real param. Compiles (all non-cloud targets); sqlite (33), sync (11), operator (49) pass. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Replace the untyped operator_os/operator_arch strings with dedicated OperatorOs / OperatorArch enums in alien-core::sync, and detect via OperatorOs::detect() / OperatorArch::detect() in the operator. Addresses the reviewer's "use an existing enum" note: the catalog's `Architecture` spells arm as `arm64` (≠ the `aarch64` std::env::consts reports) and `BinaryTarget` models buildable targets, not the host the operator runs on — so neither round-trips cleanly. The enums serialize to the exact std::env::consts spellings and match the platform's existing operatorOs/operatorArch schema 1:1. Unsupported hosts report None. The manager keeps its String request DTO (receives the serialized value), so the change stays on the wire + operator detection. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Alon asked why we chose --atomic. Document the rationale inline: --atomic rolls back a failed upgrade (that rollback is the Apply-phase failure we report), --wait gates "done" on the new pod passing its readiness probe (first successful /v1/sync), and --reuse-values keeps existing config while only image.tag flips. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- min_supported_version was set to the target version, which (once the operator enforces it) would make every operator that actually needs the upgrade refuse it — the sub-target operators are exactly the ones told to upgrade. Use a real "1.0.0" floor; raise only for stepping-stone upgrades. - is_plausible_semver split on the first of '-' or '+', so a valid pin with BOTH a pre-release and build suffix (1.4.0-rc.1+build.123) was rejected because '+' fell into the pre-release chunk. Peel '+build' first, then '-prerelease'. Added tests covering both suffixes and garbage input. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The upgrade Jobs the operator spawns carried hardcoded brand into the
customer's cluster: app.kubernetes.io/managed-by="alien-agent" and
alien.dev/{component,target-version,attempt}. A white-labeled distribution
would leak "alien" there, and the labels didn't match the chart's
white-labeled resources.
Option A: stamp the standard app.kubernetes.io/* set instead
(name=operator, instance=<KUBERNETES_HELM_RELEASE> — the chart-injected
vendor identity, component=upgrader, managed-by=operator) on both the Job
and its pod template, and select on those standard labels. The operator's
own bookkeeping (attempt, target version) moves to un-prefixed keys
(upgrade-attempt / upgrade-target-version) — no branded domain. Also fixed
stale chart comments that named a hardcoded alien-agent-upgrader SA (it's
release-derived). New test asserts no alien.dev/alien-agent leaks.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The self-update module hand-rolled reqwest calls to the in-pod Kubernetes API (build client + SA token + GET/POST) to create and list the upgrader Jobs and list pods — the only place in the tree that talks to the K8s API by hand. Route it through alien-k8s-clients instead (JobApi::create_job/list_jobs, PodApi::list_pods via KubernetesClient + try_incluster in-pod config), as the reviewer asked. Auth/transport/error handling now come from the shared client. The pure Job-body build + parse logic (and their tests) are kept, bridged through JSON, so behavior is unchanged; the benign-409 race is preserved via the error's http_status_code. New test guards that the hand-built Job body stays schema-compatible with the typed k8s-openapi Job. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…-update-for-the-alien-agent-across-all-pull-mode-2
OS-agnostic core of the operator self-update flow for the os-service
packaging (the health-gated binary swap driven by a supervising
launcher), across four crates:
- alien-core: shared launcher<->operator handoff protocol in a new
self_update module (pending/failure marker shapes, semver Version,
atomic marker IO, shared retry backoff, env + exit-code contract);
operator_target.binary is now per-host resolved {url, sha256,
signature, minLauncherVersion} — one digest cannot cover an artifact
map — and SyncRequest reports launcherVersion (reported, never
driven: the launcher is frozen and only changes via redeploy)
- alien-launcher (new crate): platform-blind supervisor core — on-disk
startup classification that recovers from a crash at any step,
swap/promote/rollback state machine with a crash-injection test
matrix, probation health gate, watchdog heartbeat thread that ticks
through probation, crash-respawn backoff, and a blocking ureq
/readyz probe. Platform shims (systemd/launchd/SCM) land next; a
guard test keeps the core free of platform code
- alien-operator: binary self-update actuator (streamed download with
sha256-while-streaming, disk preflight, staging, atomic pending.json,
graceful exit with the handoff code; ed25519 verification behind the
enforce-signature feature, failing closed on the placeholder key),
marker-based failure reporting + exponential backoff,
min_supported_version floor, /readyz extended to all five health
conditions, ALIEN_HEALTH_ADDR bind override
- alien-manager: release-manifest loader (releases_url base; http,
file://, or plain path; cached), per-host binary target emission
gated on the reported launcher version vs minLauncherVersion
("redeploy required" when below), launcher_version inventory column,
pin validation on the semver crate (rejects +build metadata and
downgrades below the reported version), admin route renamed to
PUT /v1/deployments/{id}/target-operator-version and registered in
the OpenAPI doc
Generated artifacts refreshed via `pnpm run generate:manager-openapi`
and `pnpm run generate:manager-rust-sdk`. `generate:manager-api` (TS)
is blocked by a pre-existing spec validation issue (info.license.name
is empty), unrelated to this change.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Wire the platform-blind launcher core to a real Linux service and prove the whole os-service self-update flow end-to-end. alien-launcher: - Linux ServiceHost via sd-notify (READY/WATCHDOG/STOPPING, no-op without a notify socket; watchdog interval from WATCHDOG_USEC/3) with a signal-hook stop channel drained by poll_control. - Unix ChildSupervisor (shared Linux/macOS) over command-group: spawns the operator in its own process group, sets PR_SET_PDEATHSIG on Linux so the operator dies with the launcher, graceful SIGTERM-then-SIGKILL stop. - Unix VersionStore with real symlink pointers (atomic rename flips) and statvfs disk preflight; the full phase-0 state-machine suite (promote/rollback/classification/crash-injection) now runs parameterized against this real store, not just the stub. - Synchronous main: flag parsing, platform selection, run loop; loud stub on non-Linux until those shims land. ureq built without TLS (loopback probe only) so the safety-net binary stays tiny and cross-compiles. alien-deploy-cli: - State-preserving launcher installer: idempotent version-store layout (state/ + secrets never clobbered, binaries always refreshed via temp+rename), Type=notify systemd unit from the doc-12 template, and `uninstall --purge`. Legacy direct-operator service kept behind --no-launcher (still used by `alien-deploy up`). alien-operator: - test-hooks feature: ALIEN_OPERATOR_FAKE_VERSION overrides the reported version for E2E, compile_error-guarded against release builds. Adds `operator --version`. alien-manager: - Surface launcher_version on the deployment read response (persisted + gated since phase 0 but never exposed) for the "redeploy required" signal. OpenAPI spec + Rust SDK regenerated. alien-test: - os-service E2E harness (real launcher + operator + in-process manager, artifacts served over HTTP, per-version wrapper-script identities) and 7 scenarios: promote, rollback + backoff, digest mismatch, mid-probation crash recovery, die-with-parent, and the min-launcher gate. All green on Linux. TestManager gains a releases-url override. ci: e2e-os-service.yml runs the launcher tests, the 7 scenarios, and a systemd smoke on Linux (PR path filter + nightly). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The launcher signalled systemd READY=1 only after the operator passed /readyz — which includes a successful manager sync. So if the manager was unreachable at boot (e.g. a fleet reboot during a manager outage), /readyz stayed 503, the Type=notify start timed out at 90s, and the launcher service flapped under Restart=always. That contradicts the "rollback != reachability" principle: a 503 caused by manager unreachability is not a launch failure. Split health by manager-dependence: - /livez = the local conditions (DB open, lock held, deployment loop progressing) — "up and running, possibly just waiting for the manager." No longer an unconditional 200. - /readyz = /livez + first-sync (the manager round-trip) — a strict superset, unchanged meaning. The launcher now gates its startup READY on /livez (the supervisor is operational once it has a live operator; a genuinely broken operator that crashes or never serves /livez still fails the start), and keeps the update probation gate on /readyz (a new version must prove full health, including reaching the manager, before promote discards the rollback target). This mirrors the Kubernetes livez/readyz split, with the launcher playing the kubelet's role. Also fix the CI systemd smoke: the stub operator now serves /livez, and failures dump journalctl instead of a dead artifact-upload path. All 7 os-service E2E scenarios pass on Linux; operator + launcher unit tests green. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Phase-0/1 added launcher_version to ReconcileData and an argument to update_agent_metadata, but the manager's integration-test targets (tests/*.rs) still used the old shapes and failed to compile. They were missed because `cargo test -p alien-manager --lib` builds only unit tests, not the tests/ targets. Fix the ReconcileData constructors and update_agent_metadata calls, and assert launcher_version round-trips in the full-inventory store test. (Pre-existing alien-helm insta snapshot failures on this branch are unrelated and untouched.) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…launcherVersion Add the operator `launcherVersion` field to the deployment and sync schemas in the generated Alien Platform API OpenAPI spec (and the Rust SDK copies). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ith-parent Phase 2 of the OS-service self-update: the launcher and operator now run on macOS over the shared Unix code, with only the host shim differing. - launcher: macOS launchd ServiceHost (signals + no-op notify), aliased with the shared Unix child supervisor + version store; run_supervisor enabled for macOS. All 51 launcher tests green on macOS with zero test forks. - operator die-with-parent: replace PR_SET_PDEATHSIG (absent on macOS) with a getppid() watch. Runs on a DEDICATED OS THREAD, not a tokio task — a tokio timer is starved under load and left orphans alive for minutes; a std::thread sleep is immune. 1s interval; InstanceLock remains the hard backstop. - healthy_reset: extract the crash-backoff reset into a pure, unit-tested helper and remove dead StubChild scaffolding it was meant for. - e2e: the os-service suite now runs on macOS too (6/6 green); orphans reap in <=2s at full parallelism after the OS-thread fix. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…et floor
When an admin pins an os-service operator target whose release manifest needs a
newer launcher than the frozen one installed, the manager withholds the target.
Previously this was silent; now the manager computes the verdict and forwards it
on the reconcile so the platform can show it.
- release_manifest: extract check_launcher_floor() -> {Ok | RedeployRequired |
InvalidFloor} as one pure helper shared by the target gate and the verdict.
- sync route: resolve_redeploy_status() maps the floor check to
(redeploy_required, min_launcher_version), inert for kubernetes / no pin /
already-converged / manifest error; computed in agent_sync and threaded
through two new ReconcileData fields on both os-service reconcile paths.
- refactor binary_operator_target to use the shared helper.
- regenerate the platform OpenAPI spec + Rust SDK schema for the two new
reconcile fields.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
An operator self-update handoff (or any operator exit) orphaned the user workload spawned by alien-runtime: the operator never told its worker runtime to stop, and the app child had no die-with-parent, so it was reparented to init and a second copy started under the swapped-in operator. Tear it down on both paths: - alien-operator: call the local worker manager's shutdown_all() during graceful shutdown, so the worker runtime kills the app child. - alien-runtime: spawn the app with kill_on_drop(true) and, on Linux, PR_SET_PDEATHSIG=SIGKILL, as crash backstops. macOS keeps kill_on_drop + graceful teardown (no pdeathsig); Windows is tracked to use a kill-on-close Job Object. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
alien-deploy deploy hardcoded no_launcher: true, so link-onboarded deployments always got the legacy direct-operator service and could never self-update, even with updates = auto. Default to the launcher layout (matching operator install); use_launcher_layout_for still gates it to Linux, and a new --no-launcher flag keeps the direct service as an escape hatch. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Deploy a real workload (alien-test-app built to a local OCI), capture the operator's app child, pin a new operator version, and assert that after the swap exactly one app remains and none was reparented to init. Adds rig helpers (deploy_test_app_workload, wait_for_status, app_pids, is_orphaned) and installs zig + cargo-zigbuild in the CI job for the workload's musl OCI build. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
CI showed the appless test deployment stays `pending`, so create_release's status-gated auto-assign can't attach the workload (the operator gets no app to spawn). Documented the concrete fix (seed the release before the deployment via a start_with_workload() variant) and marked the test #[ignore] so the suite stays green while the harness plumbing is finished. The fix it guards (shutdown_all + kill_on_drop/PR_SET_PDEATHSIG) is verified live. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…han guard The appless test deployment stayed `pending`, so create_release's status-gated auto-assign couldn't attach the workload. Publish the alien-test-app release BEFORE the deployment is created (new start_with_workload rig variant) so create_deployment attaches it directly via get_latest_release; the operator then spawns the app. Removes the #[ignore] and the wait_for_status / deploy_test_app_workload dance. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Implement WindowsHost, the ServiceHost boundary over the Service Control Manager. Control signals (Stop/Shutdown) arrive on a control cell drained by poll_control; report_ready/report_stopping/heartbeat drive the SCM status (Running / StopPending / checkpoint-while-pending, which the SCM ignores once Running). A --console Ctrl-C path (driven by the E2E suite) makes the status calls no-ops. Pure map_service_control + unit test. Adds windows-service/windows-sys under cfg(windows). The main.rs SCM dispatch (service_dispatcher -> core::run) and the ActiveHost alias land with T3.2 (Job-Object ChildSupervisor) and T3.4 (junction VersionStore), which core::run needs; until then the module is allow(dead_code). Cross-compiles clean for x86_64-pc-windows-gnu; host build + 52 launcher tests unaffected. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Spawn the operator in its own Job Object (`command-group`, kill-on-close = die-with-parent) and stop it with CTRL_BREAK to its process group, escalating to Job termination after the grace window. Mirrors `unix_child.rs`. Three `#[cfg(windows)]` real-process tests (stop escalation, exit-code passthrough, kill-on-close) run in the T3.7 Windows CI; here they are cross-compile-verified. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…hase 3 T3.3 Two die-with-parent gaps for the workload the operator spawns via alien-runtime: - Windows has no pdeathsig, so `AppChild` (a new per-OS wrapper) places the app in its own kill-on-close Job Object via `command-group`; the app dies when the operator exits or crashes. Linux keeps `PR_SET_PDEATHSIG`; macOS keeps `kill_on_drop`. The wrapper hides the per-OS child type behind one API so the runtime handles a single child. - The operator, as a launcher child on Windows, now honors CTRL_BREAK (the launcher's graceful-stop signal) alongside CTRL_C — draining the InstanceLock, DB, and app child. The `--service` (no-launcher SCM) path is unchanged. Also gate the operator's `Duration` import `#[cfg(unix)]` (only the Unix parent-death watch uses it), removing an unused-import warning on Windows. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…hase 3 T3.4 `current`/`last-stable` are directory junctions to `versions\<v>`. The flip is non-atomic (Windows can't rename over a dir): create `.new` -> remove -> rename. `current()` reconciles any crash residue on the next start — both-present or mid-rename finish the flip; a missing `current` is reconstructed from `probation.new` (else `last-stable`), logged loudly. `gc` tolerates a locked binary (a just-swapped-away operator .exe) by scheduling reboot-delete (MoveFileExW) instead of failing; free-space preflight via GetDiskFreeSpaceExW. Everything non-pointer delegates to `store_common`, identical to the Unix store. Tests (cross-compile-verified; run in T3.7 Windows CI): flip storm, all five crash-residue cases, locked-binary gc, and the full Phase-0 state-machine suite via `TestStoreOps` — zero suite edits. The Unix concurrent-reader test has no Windows analogue on purpose: the junction flip is non-atomic and the launcher reads/flips on one thread, so the residue tests cover interruption instead. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…se 3 T3.5 `operator install` now uses the launcher-supervised layout on Windows too (`use_launcher_layout_for` → linux|windows; `--no-launcher` still forces legacy). - The layout is cross-platform: `current`/`last-stable` are directory junctions on Windows (absolute targets), and binary names use `EXE_SUFFIX` (Linux stays byte-identical; Windows gets `alien-operator.exe` / `alien-launcher.exe`). - `install_launcher_service` refactored to a shared stop + `write_layout`, then a per-OS `register_and_start`: systemd on Linux, SCM on Windows. Windows registers `alien-launcher.exe` via service-manager (`sc create`) with the operator config passed as the service environment, then applies the doc-12 recovery config via `sc.exe failure` (reset 24h, three 5s restarts) — service-manager can't set restart policy through `sc create`, so it passes `RestartPolicy::Never` and `sc failure` is the real config. - Uninstall is unchanged (label-based); `--purge` still removes the data dir. Tests: decision matrix (windows→true), a doc-12 `sc failure` golden, and a Windows layout-idempotency test (junctions + `.exe`, mirrors the Unix one). Host tests pass; the Windows path is cross-compile-verified (runs in T3.7). Note: the launcher's Windows run_supervisor must set operator_binary to `alien-operator.exe` (EXE_SUFFIX) to match what this installer writes. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
… T3.1 Completes T3.1 — main.rs now runs the launcher as a real Windows service. - SCM mode: main() parks the parsed Args and hands control to service_dispatcher; service_main constructs WindowsHost::service(), runs the core loop, and reports the stop status (a nonzero code trips the doc-12 SCM recovery config, the Windows analogue of systemd respawning on a nonzero exit). - Console mode (--console): bypasses the dispatcher with WindowsHost::console() (Ctrl-C handler) and runs the core loop directly — what the E2E suite drives. - platform/mod.rs aliases WindowsChildSupervisor/WindowsVersionStore as ActiveChildSupervisor/ActiveVersionStore; the host is named directly since its service()/console() constructors differ from the Unix new(). - operator_binary uses EXE_SUFFIX (alien-operator.exe on Windows) to match what the T3.5 installer writes. - Drops the three #![allow(dead_code)] now that the shims are live; report_stopped takes an exit code for the SCM failure->recovery path. --console is a flag (not a `run` subcommand) — consistent with the flag-only CLI and the T3.5 service binPath. Cross-compile-verified; real SCM/console behavior runs in the T3.7 Windows CI. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…hase 3 T3.7 Add windows-os-service to e2e-os-service.yml (depot-windows-2025-16): - Build the launcher; run `cargo test -p alien-launcher -p alien-operator` on native Windows — exercising the junction VersionStore (flip + all five crash-residue cases + locked-file gc), the full Phase-0 state-machine suite over it, the Job-Object kill-on-close ChildSupervisor, and the SCM control mapping. - SCM smoke: sc create the launcher as a service, apply + assert the doc-12 recovery config (sc failure / qfailure), start -> poll RUNNING (a compiled Rust stub operator answers /livez so the launcher gates ready), stop -> poll STOPPED (the stop propagates through the launcher to the operator), delete. Covers phase exit criteria 1/2/4 and the SCM-smoke half of 3. The full seven-scenario E2E is deferred with the os_service rig port (the rig is Unix-only: symlinks, /bin/sh version wrappers, pgrep/ps/kill, a musl workload). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…ast-fail `not_ready_on_connection_refused` required a refused connection to resolve in <1s — a Unix property (the OS RSTs immediately). On Windows a connect to a just-released loopback port runs to the connect timeout before failing, so the new Windows CI (T3.7) tripped on it (took ~2.0s). Assert the probe's real contract instead: a refused connection resolves to not-ready within `timeout_global` (+1s slack) — holds cross-platform, still catches a never-returns regression. Behaviour (`!is_ready`) is unchanged and already passed on Windows. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
yonatan-alien
changed the title
…h overlay The chart's secret.yaml guardrail (`required` management.token or existingSecret) rejects partial values documents. `log_collector_enabled_chart_lints_and_templates` passed logCollector-only values and was missed in f6db604, which fixed the same guardrail failure for the manager_only render tests. Render it with the standard manager-fetch overlay like its siblings. `alien-helm --lib` is 15/15 green again. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Conflicts resolved: - alien-manager/routes/deployments.rs: keep our set_target_operator_version route + validate_target_pin; fuse both `#[cfg(test)] mod tests` (our target-pin tests + main's stack-settings/machines tests) under one `use super::*`. - alien-operator/cli.rs: union the import — ours keeps ContextError, main adds ValueEnum (for its new InitialDesiredReleaseArg). - alien-test/config.rs: identical logic, took main's multi-line formatting. - client-sdks/platform (openapi specs + generated SDK): took THEIRS. Main's #120/#121 added platform types (NewDeploymentRequestInitialDesiredRelease, KubernetesClusterSource, DeploymentSetupConfigPublicSubdomain, ...) that the auto-merged alien-cli source now depends on, so the previous "keep ours" no longer compiles. The branch's platform-schema additions (rejoin / target-agent-version) must be re-added by a platform-side openapi regen. Silent semantic merge fixed (auto-merged textually, didn't compile): - alien-manager sqlite/command_registry.rs test fixture: added the 7 new DeploymentRecord fields (our launcher_version + main's operator_arch / operator_os / operator_image_repository / operator_version / packaging / target_operator_version). Verified: ci-fast-equivalent `cargo check --workspace --tests` clean (excluding the cloud-client crates ci-fast excludes). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…ker snapshots The chart values gained the self-update `upgrade:` block, /livez+/readyz probes, persistence-enabled-by-default, and generated-encryption-key defaults, but these two insta snapshots were never regenerated — CI Fast has been red on them since before this branch's Windows/merge work. Accept the current (correct) rendered output. `cargo test -p alien-helm` is green. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Closes ALIEN-59. Manager-driven agent self-update on Kubernetes, plus the two production-hardening fixes that came out of live e2e.
Feature
agent_upgradeloop inalien-agentconsumesagent_target.helmon/v1/syncand spawns a Helm-runner Job that runshelm upgrade --reuse-values --set runtime.image.tag=<target>under analien-agent-upgraderServiceAccount.alien-helmchart guardrails —required management.{url,name,token,deploymentId},/livez+/readyzprobes, Recreate strategy, namespace-scoped upgrader Role/Binding,upgrader.enabledopt-out.agent_version/agent_os/agent_arch/regime/agent_image_repository; K8s worker transport set tolocal; K8s ServiceAccount controller is skipped (chart owns it).alien-managerexposesPUT /v1/deployments/:id/target-agent-version;build_agent_targetemitsagent_target.helmwhen target ≠ reported and regime = kubernetes.Production hardening (would silently break otherwise; both found via live e2e)
fix(helm): enable agent runtime-data persistence by default— without a PVC, the self-update's pod roll wipes the agent'sdeployment_id/sync token, the new pod hits 409 from initialize, CrashLoopBackOffs, and helm--atomicrolls back. Documented opt-out.feat(agent,manager): /v1/rejoin endpoint— for the case where persistence is genuinely off (or the state directory is wiped for any other reason), the agent's initialize-409 falls through to a newPOST /v1/rejointhat re-acquires a deployment-scoped sync token over the existing row. NewRouterOptions::include_rejoin+skip_rejoin()builder lets multi-tenant embedders override, same pattern as/v1/initialize.fix(agent): preserve 409 from initialize so rejoin fall-through fires— the agent's.context(ConfigurationError)was clobbering the SDK's real 409 with a hardcoded 500. NewDeploymentNameAlreadyExistsvariant (409) lets the conflict surface so the rejoin branch actually triggers.Paired changes are tracked in a separate PR.
Windows OS-service self-update (launcher/operator path)
This branch also lands Phase 3 (Windows) of the OS-service self-update path —
the frozen
alien-launchersupervisingalien-operatorwith health-gated binaryswaps + last-stable rollback, now on Windows via the Service Control Manager
(Linux/systemd and macOS/launchd already on the branch). All cross-platform
launcher
core/logic is shared; only the per-OS shims are new.ServiceHost+ service entry (T3.1):platform/windows.rsmaps SCMStop/Shutdown/Interrogate ↔ the launcher's control channel;
main.rs's Windowsrun_supervisorhands control toservice_dispatcher(service mode) or aCtrl-C handler (
--console, for tests/E2E), then drives the sharedcore::run.ChildSupervisor(T3.2): spawns the operator in a kill-on-closeJob (
command-group) so it dies with the launcher; graceful stop viaCTRL_BREAK, escalating to Job termination after a grace window.places the user application in its own kill-on-close Job (a new
AppChildwrapper — the Windows counterpart to Linux
PR_SET_PDEATHSIG) so a self-updatehandoff can't orphan it, and honors
CTRL_BREAKfor a clean shutdown.VersionStore(T3.4):current/last-stableare directoryjunctions; the non-atomic flip reconciles crash residue on the next start
(finish-flip / reconstruct-from-probation);
gctolerates a locked(still-mapped) operator
.exeby scheduling reboot-deletion instead of failing.The full Phase-0 state-machine suite runs unchanged over this store.
alien-deploy operator installregistersalien-launcher.exeas the SCM service under%ProgramData%\alien-operatorwith a service recovery config (
sc failure: reset 24h, three 5s restarts);junction-seeded layout with exe-suffixed binaries;
--no-launcherstill keepsthe legacy direct-operator service.
windows-os-servicejob runs the platform unit +state-machine tests and a live SCM smoke (
sc create → RUNNING → qfailure → stop → delete, with a stub operator answering/livez) on native Windows.Deferred (tracked in the internal plan): Authenticode code-signing — a
cross-platform signing workstream, since nothing is signed on any platform today
— and porting the seven-scenario E2E rig to Windows (it's Unix-only: symlinks,
/bin/shversion wrappers,pgrep/ps, a musl workload OCI build). The Windowsplatform tests + SCM smoke exercise the same store/child/SCM logic in the meantime.
Test plan
cargo test -p alien-agent -p alien-core -p alien-deployment -p alien-helm -p alien-manager --libagent_target→ helm-upgrade Job rolls the agent pod → PVC keepsdeployment_id→ new agent reports the new versionkubectl exec rm -rf /var/lib/alien-agent/*+ bounce pod → agent logsName already exists — assuming local state was wiped, rejoining…→ recovers the originaldeployment_idand stays runningclient-sdks/platform/*openapi spec + generated SDK — the v1.13.2 main merge took main's version, because main's feat: machines setup handoff and endpoint status updates #120/fix: update cli platform request fields #121 added types (NewDeploymentRequestInitialDesiredRelease,KubernetesClusterSource,DeploymentSetupConfigPublicSubdomain, …) that the auto-mergedalien-clisource depends on (keeping ours no longer compiled). Consequence: this branch's rejoin / target-agent-version schema additions are currently dropped from the vendored spec — the reverse of the earlier merge. The/v1/rejoin+ target-operator-version manager-side code and the manager SDK are intact; only the vendored SDK mirror is affected. A subsequent openapi regen must reconcile so both this branch's additions and main's land in the vendored SDK.windows-os-servicejob) green on native Windows:cargo test -p alien-launcher -p alien-operator(junction store + Phase-0 state-machine suite + Job-Object kill-on-close + SCM control mapping) and a live SCM smoke (sc create → RUNNING → sc qfailure → sc stop → sc delete) — run 29160469336