Drain sessions before replacing runtime Pods

[kelos-bot]

What type of PR is this?

/kind feature

What this PR does / why we need it:

Session StatefulSets roll out runtime image changes automatically, which can replace a Pod while it is processing a turn. Kelos must keep the StatefulSet at the desired runtime configuration so failed Pods recover correctly, while allowing healthy Pods to finish accepted work before replacement.

This change coordinates runtime replacement through Session annotations while keeping the StatefulSet at the desired configuration:

• Session StatefulSets use OnDelete, and the controller records the desired runtime configuration immediately
• the controller uses StatefulSet revisions to identify stale Pods without comparing admission-rewritten Pod fields
• a healthy stale runtime stops accepting new turns, finishes all accepted work, and reports whether it is still draining or has drained
• after a matching request ID and Pod UID report that the runtime is drained, the controller deletes that exact Pod so the StatefulSet recreates it from the desired template
• a stale Pod that is already failed is replaced immediately because it cannot reliably complete the drain protocol
• each runtime receives name-scoped RBAC for reading, watching, and patching its own Session; the controller receives Pod deletion permission

The request travels entirely through the Kubernetes API; the controller does not exec into Session Pods. The protocol does not add fields to the Session CRD.

Which issue(s) this PR is related to:

N/A

Special notes for your reviewer:

Validation:

• make update
• make verify
• make test
• make test-integration (125 specs)
• make build

The E2E suite holds a turn at a user-input request, verifies that the desired StatefulSet is current while the stale Pod remains in place, then verifies that a new Pod is created only after the turn drains.

Does this PR introduce a user-facing change?

Kelos now waits for active Session work to drain before replacing runtime Pods during upgrades, while failed stale Pods are replaced immediately from the desired runtime configuration.

A Session Pod created by a Kelos version that predates runtime draining requires one explicitly forced, known-idle update. After confirming that the Session is idle, run `kubectl annotate session NAME kelos.dev/force-session-runtime-update=true`.

[gjkim42]

/kelos review

[kelos-bot]

🤖 Kelos Task Status

Task kelos-reviewer-issue-comment-09b18e7bb4fc has succeeded. ✅

[kelos-bot]

🤖 Kelos Reviewer Agent @gjkim42

Review Summary

Verdict: APPROVE
Overall correctness: patch is correct
Scope: Coordinates Session runtime replacement with OnDelete StatefulSets, revision-based stale-Pod detection, and an annotation-based drain handshake with UID-safe deletion.

Findings Overview

Priority	Count	File:Line	Summary
P0	0	—	none
P1	0	—	none
P2	1	internal/controller/session_runtime_access.go:95	Runtime token can patch unrelated Session metadata
P3	0	—	none

Findings

Security

• [P2] internal/controller/session_runtime_access.go:95 — Avoid granting the runtime patch on the entire Session object solely to write the update-report annotation. The service-account token is mounted in the agent container, where custom provider images and repository-controlled code execute, and Kubernetes RBAC cannot constrain a patch to one annotation; that code can therefore add a finalizer or alter any other mutable metadata on its named Session, making deletion hang until an administrator repairs it. Store the runtime acknowledgement on a field-scoped surface or enforce the allowed annotation-only metadata diff in admission.

Key takeaways

• StatefulSet revision matching, request ID and Pod UID validation, and the Pod deletion UID precondition make the replacement path robust against admission rewrites and stale acknowledgements.
• The release note's known-idle force annotation provides the explicit compatibility path for runtimes that predate the drain protocol.
• The native codex review path verified the target commit but could not inspect the diff because its sandbox failed during loopback-network initialization; the independent Google-rubric review and direct project-specific review completed.

[gjkim42]

/kelos review

[kelos-bot]

🤖 Kelos Task Status

Task kelos-reviewer-issue-comment-f222788fc6f9 has succeeded. ✅

[gjkim42]

/kelos review

[kelos-bot]

🤖 Kelos Task Status

Task kelos-reviewer-issue-comment-7cb2c486457b has succeeded. ✅

[kelos-bot]

🤖 Kelos Agent @gjkim42

Addressed P1 in the squashed branch update. Session Pod currency uses the StatefulSet controller revision after the StatefulSet controller has observed the current generation, so admission-rewritten runtime images do not trigger repeated drain and replacement. The unit regression covers a digest/mirror rewrite, and the integration/e2e fixtures model StatefulSet revisions.

P2 is intentionally declined per maintainer direction. The forced known-idle update is a one-time migration procedure for Sessions created before runtime draining existed, so it remains in this PR's release-note block only and is intentionally not duplicated in docs/reference.md or the Helm README.

[gjkim42]

/kelos review

[kelos-bot]

🤖 Kelos Task Status

Task kelos-reviewer-issue-comment-b4a4a35826db has succeeded. ✅