Pull-based job agents V2: lease, heartbeat, and crash recovery

[adityachoudhari26]

Follow-up to the V1 pull-based job agent work (#1157, RFC 0014). V1 ships the happy path — an external agent lists its queued jobs and atomically claims one (queued → in_progress). V2 adds crash recovery so a job whose agent dies mid-execution becomes reclaimable instead of stuck in_progress.

This addresses failure mode #2 from the original issue (#1153): "crash mid-job." V1 deliberately deferred it — today a crashed agent leaves its job stuck in_progress, recoverable only by a manual status override back to queued. V2 automates that.

Design (per RFC 0014)

Everything is additive: a new table, two endpoints, and a periodic sweep. The job table is not modified — queued already shipped in V1.

job_claim table

Lease state lives in a dedicated table rather than as columns on job. The motivation is write locality: heartbeats are the highest-frequency write in this feature (every in-flight job, every interval), and job is a hot, heavily-joined table. Keeping heartbeat writes off it avoids index churn and MVCC bloat on the read path.

CREATE TABLE job_claim (
  job_id           uuid PRIMARY KEY REFERENCES job(id) ON DELETE CASCADE,
  job_agent_id     uuid NOT NULL,
  claimed_at       timestamptz NOT NULL DEFAULT now(),
  claim_expires_at timestamptz NOT NULL,
  claim_id         uuid NOT NULL DEFAULT gen_random_uuid()
);

job.status remains the state machine (the claim still flips queued → in_progress); only the lease lifecycle lives in job_claim.

Claim records the lease

The claim flips status and records the lease in one atomic statement (CTE):

WITH claimed AS (
  UPDATE job SET status = 'in_progress', started_at = now()
  WHERE id = (
    SELECT id FROM job
    WHERE status = 'queued' AND job_agent_id = $1
    ORDER BY created_at LIMIT 1
    FOR UPDATE SKIP LOCKED
  )
  RETURNING id
)
INSERT INTO job_claim (job_id, job_agent_id, claim_expires_at)
SELECT id, $1, now() + make_interval(secs => $lease_seconds) FROM claimed
RETURNING *;

The lease is a liveness window, not an execution deadline — a job may run far longer than the lease as long as the agent heartbeats. The claim response advertises lease_seconds so the agent picks its own heartbeat interval.

Heartbeat endpoint

POST /v1/workspaces/{workspaceId}/jobs/{jobId}/heartbeat

Extends the lease, touching only job_claim (never job):

UPDATE job_claim SET claim_expires_at = now() + make_interval(secs => $lease_seconds) WHERE job_id = $1;

Reaper

A periodic sweep returns abandoned claims to the queue — delete the expired claim and flip the job back to queued, in one statement:

WITH expired AS (
  DELETE FROM job_claim WHERE claim_expires_at < now() RETURNING job_id
)
UPDATE job SET status = 'queued'
WHERE id IN (SELECT job_id FROM expired) AND status = 'in_progress';

Mirrors the reconcile queue's CleanupExpiredClaims. Expiry is detected by the sweep, not by an event at the exact expiry time.

Reclaim is opt-in by construction: only jobs that have a job_claim row are ever swept. A job claimed without recording lease state — or any V1-era agent that never heartbeats — has no claim row and is never reclaimed, preserving V1 behavior after V2 ships. When a job reaches a terminal status, its job_claim row is removed.

Double-run note

The reaper can't distinguish a crashed agent from one that's alive but quiet longer than the lease, so a long pause can cause a job to be reclaimed and run twice. A generous lease relative to the heartbeat interval shrinks the window but doesn't close it. If exactly-once is required, the claim_id fencing token (present in the schema from the start) is echoed by the agent on heartbeat/status and a stale token is rejected. Enforcement is optional.

Scope checklist

• job_claim table + migration (and sqlc mirror)
• Claim endpoint records the job_claim row (CTE) and returns lease_seconds + claim_id
• POST .../jobs/{jobId}/heartbeat endpoint (+ OpenAPI path)
• Reaper sweep (periodic; mirrors CleanupExpiredClaims)
• Remove the job_claim row when a job reaches a terminal status
• e2e coverage (lease expiry → reclaim; heartbeat keeps a long job claimed)

Open questions (from RFC 0014)

• Lease configuration: per-agent (job_agent.config, bounded) or a single global default? Start global; per-agent later for agents with different reliability characteristics.
• Fencing: enforce the claim_id token from the start, or only if double-run under lease expiry proves to be a real problem?
• Long-poll vs plain poll for the V1 list/claim path (reduces idle polling; backpressure/fairness limits on held connections).

Refs: RFC docs/rfc/0014-pull-based-job-agent.mdx, V1 #1157, original #1153.