diff --git a/CHANGELOG.md b/CHANGELOG.md index 846ae76..bd9d33b 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,6 +6,9 @@ All notable changes to Tabellio are recorded here. ### Added +- Provider-neutral exact-commit validation runner with committed argv manifests and no shell execution. +- Bounded SHA-256 output evidence, detached worktree cleanup, and durable results on `refs/tabellio/validations`. +- Local validation results integrated into durable review readiness. - Git-native JSON ledger on compare-and-swap refs for durable control-plane state. - Durable review cycles covering Forgejo feedback, provider-neutral agent findings, triage, checks, fixes, and readiness. - Entire-checkpoint-bound fix records with commit remapping across git-spice restacks. diff --git a/README.md b/README.md index 6e0bae3..81c761d 100644 --- a/README.md +++ b/README.md @@ -86,6 +86,7 @@ GitHub remains an optional storage, CI, and review surface. Entire is the defaul | `scripts/providers/forgejo-provider.mjs` | Read-only Forgejo repository and review adapter | | `scripts/lib/git-json-ledger.mjs` | Versioned, compare-and-swap JSON state on standard Git refs | | `scripts/lib/review-cycle.mjs` | Durable forge and agent review/fix state machine | +| `scripts/lib/validation-runner.mjs` | Exact-commit, shell-free validation with bounded evidence logs | | `infra/forgejo/` | Disposable localhost Forgejo integration lab | | `scripts/lib/` | Git process, repository contract, worktree, and context primitives | | `scripts/` | Dependency-free capture, writer, and validators | @@ -177,6 +178,7 @@ npm run tabellio:stack -- --repo . --repo-id IntelIP/Tabellio --out tabellio-sta npm run tabellio:stack:check npm run tabellio:stack:operation:example:check npm run tabellio:review:example:check +npm run tabellio:validate:example:check npm run tabellio:ledger -- --repo . --repo-id IntelIP/Tabellio --base main --head HEAD --out tabellio-ledger.json npm run tabellio:ledger:check npm run tabellio:forge -- version --base-url http://127.0.0.1:3300 --token-file .tabellio/forgejo/credentials/admin-token @@ -209,6 +211,7 @@ The external-action checker fails when an action is marked `attempted: true` wit - [Agent run lifecycle](docs/agent-run-lifecycle.md) - [Approved stack operations](docs/stack-operations.md) - [Durable review and fix loop](docs/review-loop.md) +- [Provider-neutral validation](docs/validation-runner.md) - [Workflow model](docs/workflow-model.md) - [Native Git foundation](docs/native-git-foundation.md) - [Evidence schema](docs/evidence-schema.md) diff --git a/docs/review-loop.md b/docs/review-loop.md index 1239a40..708cddd 100644 --- a/docs/review-loop.md +++ b/docs/review-loop.md @@ -24,7 +24,7 @@ Review status is deterministic: | `changes_requested` | Actionable feedback remains open | | `update_required` | Fix exists locally but is not in the remote PR head | | `blocked` | A remote check failed or the forge reports the change as non-mergeable | -| `validating` | Checks are pending or running | +| `validating` | No validation result exists yet, or checks are pending/running | | `ready` | Feedback is handled, fixes are published, and checks are clear | | `merged` / `closed` | Forge terminal state | @@ -41,7 +41,7 @@ node scripts/tabellio-review.mjs sync \ --actor review-sync-agent ``` -Sync imports reviews, inline review comments, issue comments, and commit checks. Missing provider items are retained as stale evidence rather than silently deleted. +Sync imports reviews, inline review comments, issue comments, forge commit checks, and the newest provider-neutral Tabellio validation for the PR head. Missing provider items are retained as stale evidence rather than silently deleted. A PR with no validation remains `validating`. ## Import A Codex Review diff --git a/docs/tooling-stack.md b/docs/tooling-stack.md index 18fa037..d26d571 100644 --- a/docs/tooling-stack.md +++ b/docs/tooling-stack.md @@ -75,6 +75,7 @@ Included: - read-only Forgejo provider for repositories, pull requests, reviews, comments, and commit statuses - approval-gated git-spice submit, update, sync, restack, and merge operations with one-use receipts - Git-native review ledger with Forgejo feedback, provider-neutral agent findings, triage, fixes, and readiness state +- provider-neutral exact-commit validation runner with durable results on `refs/tabellio/validations` Not included yet: diff --git a/docs/validation-runner.md b/docs/validation-runner.md new file mode 100644 index 0000000..5c1f56a --- /dev/null +++ b/docs/validation-runner.md @@ -0,0 +1,64 @@ +# Provider-Neutral Validation + +Tabellio validation replaces the GitHub Actions runtime dependency with a small local contract. Any trusted worker that has Git, Node.js, the repository, and its declared tools can run the same manifest. Results live in standard Git objects on `refs/tabellio/validations`. + +## Manifest + +`tabellio.validation.json` is committed with the code being tested: + +```json +{ + "schemaVersion": "tabellio-validation/v0.1", + "id": "tabellio-default", + "failFast": true, + "requireEntireCheckpoint": true, + "commands": [{ + "id": "repository-check", + "argv": ["npm", "run", "check"], + "cwd": ".", + "timeoutMs": 1200000, + "required": true + }] +} +``` + +Commands are executable-plus-argument arrays. Tabellio never invokes a shell, interpolates command strings, or reads a newer working-tree manifest while validating an older commit. + +## Run + +```bash +node scripts/tabellio-validate.mjs run \ + --repo . \ + --repo-id example/project \ + --base main \ + --commit HEAD \ + --manifest tabellio.validation.json \ + --runner-id worker-01 +``` + +The runner: + +1. Freezes base and head object IDs and computes the merge base. +2. Reads the manifest from the exact head commit. +3. Requires at least one Entire checkpoint in the change range when configured. +4. Creates a detached temporary worktree at the exact head commit. +5. Runs argv commands with an isolated `HOME`/`TMPDIR`, minimal environment, and per-command timeout. +6. Hashes all output while retaining only the newest 16 KiB of each stream. +7. Removes the worktree even after failure. +8. Writes an integrity-protected result to `refs/tabellio/validations` with compare-and-swap retries. + +Read the newest result for a commit: + +```bash +node scripts/tabellio-validate.mjs latest --repo . --commit HEAD +``` + +## CI Worker Shape + +The runner is deliberately scheduler-agnostic. A Forgejo webhook worker, systemd service, AWS Batch task, Kubernetes Job, Buildkite agent, Woodpecker pipeline, or local Codex worker can invoke the same CLI. The scheduler supplies capacity; Tabellio supplies the immutable validation contract and result ledger. + +Publishing `refs/tabellio/validations` to a remote is a separate approved Git write. Review sync consumes the latest result for the exact PR head and will not reuse a result from an older commit. + +## Security Boundary + +Detached worktrees and isolated home directories protect host source and common credential paths; they are not a hostile-code sandbox. Validation commands can execute arbitrary repository code and use the network. Run untrusted contributions in a disposable VM, container, or sandbox with scoped credentials and network policy. Never place secrets in manifest arguments or print them: argv and bounded output tails are retained as durable evidence. diff --git a/examples/tabellio-validation/minimal-result.json b/examples/tabellio-validation/minimal-result.json new file mode 100644 index 0000000..c9019ca --- /dev/null +++ b/examples/tabellio-validation/minimal-result.json @@ -0,0 +1,49 @@ +{ + "schemaVersion": "tabellio-validation-result/v0.1", + "runId": "validation-example-001", + "repository": { "id": "example/tabellio" }, + "revision": { + "baseCommit": "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa", + "mergeBase": "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa", + "headCommit": "bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb" + }, + "suite": { + "id": "tabellio-default", + "manifestPath": "tabellio.validation.json", + "manifestDigest": "cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc" + }, + "runner": { "id": "example-runner", "runtime": "node-v20.0.0" }, + "status": "passed", + "checkpoints": ["checkpoint-example-001"], + "commands": [{ + "id": "repository-check", + "argv": ["npm", "run", "check"], + "cwd": ".", + "required": true, + "status": "passed", + "exitCode": 0, + "signal": null, + "durationMs": 1000, + "stdout": { + "bytes": 14, + "digest": "0c7f9ceb9b131d26399e998396e479d5d9e46d67c2819faef8d2ea184e63424b", + "tail": "checks passed\n", + "truncated": false + }, + "stderr": { + "bytes": 0, + "digest": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", + "tail": "", + "truncated": false + }, + "startedAt": "2026-07-10T12:00:00.000Z", + "completedAt": "2026-07-10T12:00:01.000Z", + "error": null + }], + "startedAt": "2026-07-10T12:00:00.000Z", + "completedAt": "2026-07-10T12:00:01.000Z", + "integrity": { + "algorithm": "sha256", + "digest": "e0bf535d574ca275855e3fee7cccc4512785eaa9f4041975459d898145a4338f" + } +} diff --git a/package.json b/package.json index a0c47e9..dcd4025 100644 --- a/package.json +++ b/package.json @@ -15,6 +15,7 @@ "tabellio-stack": "scripts/tabellio-stack.mjs", "tabellio-stack-operation": "scripts/tabellio-stack-operation.mjs", "tabellio-review": "scripts/tabellio-review.mjs", + "tabellio-validate": "scripts/tabellio-validate.mjs", "tabellio-ledger": "scripts/tabellio-ledger.mjs", "tabellio-forge": "scripts/tabellio-forge.mjs" }, @@ -51,6 +52,8 @@ "tabellio:stack:operation:example:check": "node scripts/check-tabellio-stack-operation.mjs --intent examples/tabellio-stack-operation/minimal-intent.json --approval examples/tabellio-stack-operation/minimal-approval.json --at 2026-07-10T12:02:00.000Z", "tabellio:review": "node scripts/tabellio-review.mjs", "tabellio:review:example:check": "node scripts/check-tabellio-review-cycle.mjs --cycle examples/tabellio-review/minimal-cycle.json && node scripts/check-tabellio-agent-review.mjs --review examples/tabellio-review/minimal-agent-review.json", + "tabellio:validate": "node scripts/tabellio-validate.mjs", + "tabellio:validate:example:check": "node scripts/check-tabellio-validation.mjs --manifest tabellio.validation.json && node scripts/check-tabellio-validation.mjs --result examples/tabellio-validation/minimal-result.json", "tabellio:ledger": "node scripts/tabellio-ledger.mjs", "tabellio:ledger:check": "node scripts/check-tabellio-ledger.mjs --ledger tabellio-ledger.json", "tabellio:ledger:example:check": "node scripts/check-tabellio-ledger.mjs --ledger examples/tabellio-ledger/minimal-ledger.json", @@ -62,7 +65,7 @@ "tabellio:evidence:write": "node scripts/write-tabellio-evidence-envelope.mjs --out tabellio-pr-evidence.json", "tabellio:evidence:check": "node scripts/check-tabellio-evidence-envelope.mjs --evidence tabellio-pr-evidence.json", "tabellio:external-actions:check": "node scripts/check-tabellio-external-actions.mjs --evidence tabellio-pr-evidence.json", - "check": "npm test && npm run tabellio:run:example:check && npm run tabellio:stack:example:check && npm run tabellio:stack:operation:example:check && npm run tabellio:review:example:check && npm run tabellio:ledger:example:check && npm run tabellio:context:example:check && npm run tabellio:evidence:example:check && node scripts/check-tabellio-external-actions.mjs --evidence examples/tabellio-evidence/minimal-evidence.json" + "check": "npm test && npm run tabellio:run:example:check && npm run tabellio:stack:example:check && npm run tabellio:stack:operation:example:check && npm run tabellio:review:example:check && npm run tabellio:validate:example:check && npm run tabellio:ledger:example:check && npm run tabellio:context:example:check && npm run tabellio:evidence:example:check && node scripts/check-tabellio-external-actions.mjs --evidence examples/tabellio-evidence/minimal-evidence.json" }, "engines": { "node": ">=20" diff --git a/schemas/validation-manifest.schema.json b/schemas/validation-manifest.schema.json new file mode 100644 index 0000000..3d62d1c --- /dev/null +++ b/schemas/validation-manifest.schema.json @@ -0,0 +1,31 @@ +{ + "$schema": "https://json-schema.org/draft/2020-12/schema", + "$id": "urn:tabellio:schema:validation-manifest:v0.1", + "title": "Tabellio Validation Manifest", + "type": "object", + "required": ["schemaVersion", "id", "failFast", "requireEntireCheckpoint", "commands"], + "additionalProperties": false, + "properties": { + "schemaVersion": { "const": "tabellio-validation/v0.1" }, + "id": { "type": "string", "minLength": 1 }, + "failFast": { "type": "boolean" }, + "requireEntireCheckpoint": { "type": "boolean" }, + "commands": { + "type": "array", + "minItems": 1, + "maxItems": 50, + "items": { + "type": "object", + "required": ["id", "argv", "cwd", "timeoutMs", "required"], + "additionalProperties": false, + "properties": { + "id": { "type": "string", "minLength": 1 }, + "argv": { "type": "array", "minItems": 1, "maxItems": 100, "items": { "type": "string", "minLength": 1 } }, + "cwd": { "type": "string", "minLength": 1 }, + "timeoutMs": { "type": "integer", "minimum": 100, "maximum": 3600000 }, + "required": { "type": "boolean" } + } + } + } + } +} diff --git a/schemas/validation-result.schema.json b/schemas/validation-result.schema.json new file mode 100644 index 0000000..2ff80d7 --- /dev/null +++ b/schemas/validation-result.schema.json @@ -0,0 +1,96 @@ +{ + "$schema": "https://json-schema.org/draft/2020-12/schema", + "$id": "urn:tabellio:schema:validation-result:v0.1", + "title": "Tabellio Validation Result", + "type": "object", + "required": ["schemaVersion", "runId", "repository", "revision", "suite", "runner", "status", "checkpoints", "commands", "startedAt", "completedAt", "integrity"], + "additionalProperties": false, + "properties": { + "schemaVersion": { "const": "tabellio-validation-result/v0.1" }, + "runId": { "type": "string", "minLength": 1 }, + "repository": { + "type": "object", + "required": ["id"], + "additionalProperties": false, + "properties": { "id": { "type": "string", "minLength": 1 } } + }, + "revision": { + "type": "object", + "required": ["baseCommit", "mergeBase", "headCommit"], + "additionalProperties": false, + "properties": { + "baseCommit": { "$ref": "#/$defs/oid" }, + "mergeBase": { "$ref": "#/$defs/oid" }, + "headCommit": { "$ref": "#/$defs/oid" } + } + }, + "suite": { + "type": "object", + "required": ["id", "manifestPath", "manifestDigest"], + "additionalProperties": false, + "properties": { + "id": { "type": "string", "minLength": 1 }, + "manifestPath": { "type": "string", "minLength": 1 }, + "manifestDigest": { "$ref": "#/$defs/sha256" } + } + }, + "runner": { + "type": "object", + "required": ["id", "runtime"], + "additionalProperties": false, + "properties": { + "id": { "type": "string", "minLength": 1 }, + "runtime": { "type": "string", "minLength": 1 } + } + }, + "status": { "enum": ["passed", "failed"] }, + "checkpoints": { "type": "array", "uniqueItems": true, "items": { "type": "string", "minLength": 1 } }, + "commands": { "type": "array", "minItems": 1, "items": { "$ref": "#/$defs/commandResult" } }, + "startedAt": { "type": "string", "format": "date-time" }, + "completedAt": { "type": "string", "format": "date-time" }, + "integrity": { + "type": "object", + "required": ["algorithm", "digest"], + "additionalProperties": false, + "properties": { + "algorithm": { "const": "sha256" }, + "digest": { "$ref": "#/$defs/sha256" } + } + } + }, + "$defs": { + "oid": { "type": "string", "pattern": "^(?:[0-9a-f]{40}|[0-9a-f]{64})$" }, + "sha256": { "type": "string", "pattern": "^[0-9a-f]{64}$" }, + "output": { + "type": "object", + "required": ["bytes", "digest", "tail", "truncated"], + "additionalProperties": false, + "properties": { + "bytes": { "type": "integer", "minimum": 0 }, + "digest": { "$ref": "#/$defs/sha256" }, + "tail": { "type": "string" }, + "truncated": { "type": "boolean" } + } + }, + "commandResult": { + "type": "object", + "required": ["id", "argv", "cwd", "required", "status", "exitCode", "signal", "durationMs", "stdout", "stderr", "startedAt", "completedAt", "error"], + "additionalProperties": false, + "properties": { + "id": { "type": "string", "minLength": 1 }, + "argv": { "type": "array", "minItems": 1, "items": { "type": "string", "minLength": 1 } }, + "cwd": { "type": "string", "minLength": 1 }, + "required": { "type": "boolean" }, + "status": { "enum": ["passed", "failed", "error", "timed_out", "skipped"] }, + "exitCode": { "type": ["integer", "null"] }, + "signal": { "type": ["string", "null"] }, + "durationMs": { "type": "integer", "minimum": 0 }, + "stdout": { "$ref": "#/$defs/output" }, + "stderr": { "$ref": "#/$defs/output" }, + "startedAt": { "oneOf": [{ "type": "string", "format": "date-time" }, { "type": "null" }] }, + "completedAt": { "oneOf": [{ "type": "string", "format": "date-time" }, { "type": "null" }] }, + "error": { "type": ["string", "null"] } + } + } + } +} diff --git a/scripts/check-tabellio-validation.mjs b/scripts/check-tabellio-validation.mjs new file mode 100755 index 0000000..228bc90 --- /dev/null +++ b/scripts/check-tabellio-validation.mjs @@ -0,0 +1,36 @@ +#!/usr/bin/env node + +import { readFile } from "node:fs/promises"; +import { resolve } from "node:path"; + +import { validateValidationManifest, validateValidationResult } from "./lib/validation-runner.mjs"; + +const type = process.argv[2]; +const path = process.argv[3]; +const blockers = []; +let value = null; +try { + if (!path || process.argv.length !== 4 || !["--manifest", "--result"].includes(type)) { + throw new Error("Usage: check-tabellio-validation (--manifest|--result) ."); + } + value = JSON.parse(await readFile(resolve(path), "utf8")); + if (type === "--manifest") validateValidationManifest(value); + else validateValidationResult(value); +} catch (error) { + blockers.push(error instanceof Error ? error.message : String(error)); +} +const result = { + ok: blockers.length === 0, + status: blockers.length === 0 ? "validation_contract_ready" : "blocked", + checkedAt: new Date().toISOString(), + type: type === "--manifest" ? "manifest" : type === "--result" ? "result" : null, + path, + summary: value ? { + schemaVersion: value.schemaVersion, + id: value.id ?? value.runId ?? null, + commandCount: Array.isArray(value.commands) ? value.commands.length : null, + } : null, + blockers, +}; +if (!result.ok) process.exitCode = 1; +console.log(JSON.stringify(result, null, 2)); diff --git a/scripts/lib/review-cycle.mjs b/scripts/lib/review-cycle.mjs index 3674508..c630ce0 100644 --- a/scripts/lib/review-cycle.mjs +++ b/scripts/lib/review-cycle.mjs @@ -2,14 +2,16 @@ import { createHash, randomUUID } from "node:crypto"; import { runGit } from "./git-process.mjs"; import { digestObject } from "./stack-operation.mjs"; +import { latestValidationResult } from "./validation-runner.mjs"; export const REVIEW_CYCLE_SCHEMA_VERSION = "tabellio-review-cycle/v0.1"; export const AGENT_REVIEW_SCHEMA_VERSION = "tabellio-agent-review/v0.1"; export class ReviewCycleManager { - constructor({ store, ledger, provider, repositoryId, owner, repo }) { + constructor({ store, ledger, validationLedger = null, provider, repositoryId, owner, repo }) { this.store = store; this.ledger = ledger; + this.validationLedger = validationLedger; this.provider = provider; this.repositoryId = repositoryId; this.owner = owner; @@ -24,14 +26,19 @@ export class ReviewCycleManager { this.provider.listReviews({ owner: this.owner, repo: this.repo, number }), this.provider.listIssueComments({ owner: this.owner, repo: this.repo, number }), ]); - const [reviewComments, checks] = await Promise.all([ + const [reviewComments, forgeChecks, localValidation] = await Promise.all([ this.provider.listReviewComments({ owner: this.owner, repo: this.repo, number, reviews }), this.provider.commitStatus({ owner: this.owner, repo: this.repo, commit: changeRequest.source.commit, }), + this.validationLedger ? latestValidationResult(this.validationLedger, changeRequest.source.commit) : null, ]); + if (localValidation && localValidation.repository.id !== this.repositoryId) { + throw new Error("Local validation repository does not match the review cycle."); + } + const checks = mergeChecks(forgeChecks, localValidation); const record = await this.#read(number); const existing = record.value; if (existing) validateReviewCycle(existing); @@ -469,7 +476,7 @@ function deriveStatus(cycle) { if (live.some((item) => item.disposition === "pending")) return "needs_triage"; if (live.some((item) => item.disposition === "actionable" && item.resolution === "open")) return "changes_requested"; if (cycle.fixes.some((fix) => fix.published !== true)) return "update_required"; - if (["pending", "running"].includes(cycle.checks.state)) return "validating"; + if (["none", "pending", "running"].includes(cycle.checks.state)) return "validating"; return "ready"; } @@ -478,6 +485,25 @@ function cycleDigest(value) { return digestObject(unsigned); } +function mergeChecks(forgeChecks, localValidation) { + if (!localValidation) return forgeChecks; + const localStatus = { + id: `validation:${localValidation.runId}`, + context: `tabellio/${localValidation.suite.id}`, + state: localValidation.status === "passed" ? "success" : "failure", + description: `Provider-neutral validation ${localValidation.status}.`, + targetUrl: null, + createdAt: localValidation.startedAt, + updatedAt: localValidation.completedAt, + }; + const statuses = [...forgeChecks.statuses, localStatus]; + let state = forgeChecks.state; + if (localStatus.state === "failure" || ["error", "failure", "failed"].includes(forgeChecks.state)) state = "failure"; + else if (["pending", "running"].includes(forgeChecks.state)) state = forgeChecks.state; + else state = "success"; + return { commit: forgeChecks.commit, state, total: statuses.length, statuses }; +} + function cycleId(repositoryId, owner, repo, number) { return `review-${createHash("sha256").update(`${repositoryId}\0${owner}\0${repo}\0${number}`).digest("hex").slice(0, 24)}`; } diff --git a/scripts/lib/validation-runner.mjs b/scripts/lib/validation-runner.mjs new file mode 100644 index 0000000..ddf46c7 --- /dev/null +++ b/scripts/lib/validation-runner.mjs @@ -0,0 +1,412 @@ +import { spawn } from "node:child_process"; +import { createHash, randomUUID } from "node:crypto"; +import { mkdir, rm, stat } from "node:fs/promises"; +import { isAbsolute, relative, resolve } from "node:path"; + +import { LedgerConflictError } from "./git-json-ledger.mjs"; +import { runGit } from "./git-process.mjs"; +import { digestObject } from "./stack-operation.mjs"; + +export const VALIDATION_MANIFEST_SCHEMA_VERSION = "tabellio-validation/v0.1"; +export const VALIDATION_RESULT_SCHEMA_VERSION = "tabellio-validation-result/v0.1"; +const MAX_OUTPUT_TAIL_BYTES = 16 * 1024; + +export class ValidationRunner { + constructor({ store, ledger }) { + this.store = store; + this.ledger = ledger; + } + + async run({ + repositoryId, + commit, + base, + manifestPath = "tabellio.validation.json", + runnerId = "local", + now = new Date(), + }) { + requiredString(repositoryId, "repositoryId"); + requiredString(runnerId, "runnerId"); + validateRelativePath(manifestPath, "manifestPath"); + const [headCommit, baseCommit] = await Promise.all([ + this.store.resolveRef(commit), + this.store.resolveRef(base), + ]); + const mergeBase = (await runGit({ + args: ["merge-base", baseCommit, headCommit], + cwd: this.store.repoPath, + })).stdout.trim(); + const manifestSource = await runGit({ + args: ["show", `${headCommit}:${manifestPath}`], + cwd: this.store.repoPath, + }); + const manifest = JSON.parse(manifestSource.stdout); + validateValidationManifest(manifest); + const checkpoints = await checkpointIds(this.store.repoPath, mergeBase, headCommit); + if (manifest.requireEntireCheckpoint && checkpoints.length === 0) { + throw new Error(`Validation range ${mergeBase}..${headCommit} has no Entire checkpoint.`); + } + + const runId = `validation-${randomUUID()}`; + const common = await runGit({ args: ["rev-parse", "--git-common-dir"], cwd: this.store.repoPath }); + const workspaceRoot = resolve(this.store.repoPath, common.stdout.trim(), "tabellio", "validation-workspaces"); + const workspace = resolve(workspaceRoot, runId); + const home = resolve(workspaceRoot, `${runId}.home`); + await mkdir(workspaceRoot, { recursive: true }); + if (await stat(workspace).catch(() => null)) throw new Error(`Validation workspace already exists: ${workspace}.`); + const startedAt = now.toISOString(); + const commands = []; + try { + await runGit({ args: ["worktree", "add", "--detach", workspace, headCommit], cwd: this.store.repoPath }); + await mkdir(resolve(home, "tmp"), { recursive: true }); + let stopped = false; + for (const command of manifest.commands) { + if (stopped) { + commands.push(skippedCommand(command)); + continue; + } + const result = await runValidationCommand(command, workspace, home); + commands.push(result); + if (manifest.failFast && command.required && result.status !== "passed") stopped = true; + } + } finally { + await runGit({ + args: ["worktree", "remove", "--force", workspace], + cwd: this.store.repoPath, + acceptableExitCodes: [0, 128], + }).catch(() => {}); + await rm(workspace, { recursive: true, force: true }); + await rm(home, { recursive: true, force: true }); + } + const completedAt = new Date().toISOString(); + const requiredFailed = commands.some((command, index) => manifest.commands[index].required && command.status !== "passed"); + const result = { + schemaVersion: VALIDATION_RESULT_SCHEMA_VERSION, + runId, + repository: { id: repositoryId }, + revision: { baseCommit, mergeBase, headCommit }, + suite: { + id: manifest.id, + manifestPath, + manifestDigest: digestObject(manifest), + }, + runner: { id: runnerId, runtime: `node-${process.version}` }, + status: requiredFailed ? "failed" : "passed", + checkpoints, + commands, + startedAt, + completedAt, + integrity: { algorithm: "sha256", digest: "0".repeat(64) }, + }; + result.integrity.digest = validationResultDigest(result); + validateValidationResult(result); + const path = validationPath(headCommit, runId); + const written = await writeResultWithRetry(this.ledger, path, result); + return { result, path, version: written.version }; + } +} + +export async function latestValidationResult(ledger, commit) { + oid(commit, "commit"); + const prefix = `commits/${commit}`; + const listed = await ledger.list(prefix); + let latest = null; + for (const path of listed.paths) { + const record = await ledger.read(path); + if (!record.value) continue; + validateValidationResult(record.value); + if (record.value.revision.headCommit !== commit) throw new Error(`Validation result ${path} is stored under the wrong commit.`); + if (!latest || Date.parse(record.value.completedAt) > Date.parse(latest.completedAt)) latest = record.value; + } + return latest; +} + +export function validateValidationManifest(value) { + object(value, "validation manifest"); + exactKeys(value, ["schemaVersion", "id", "failFast", "requireEntireCheckpoint", "commands"], "validation manifest"); + equals(value.schemaVersion, VALIDATION_MANIFEST_SCHEMA_VERSION, "validation manifest.schemaVersion"); + requiredString(value.id, "validation manifest.id"); + boolean(value.failFast, "validation manifest.failFast"); + boolean(value.requireEntireCheckpoint, "validation manifest.requireEntireCheckpoint"); + if (!Array.isArray(value.commands) || value.commands.length === 0 || value.commands.length > 50) { + throw new Error("validation manifest.commands must contain 1 to 50 commands."); + } + const ids = new Set(); + for (const [index, command] of value.commands.entries()) { + const path = `validation manifest.commands[${index}]`; + object(command, path); + exactKeys(command, ["id", "argv", "cwd", "timeoutMs", "required"], path); + requiredString(command.id, `${path}.id`); + if (ids.has(command.id)) throw new Error(`validation manifest.commands contains duplicate id ${command.id}.`); + ids.add(command.id); + if (!Array.isArray(command.argv) || command.argv.length === 0 || command.argv.length > 100) throw new Error(`${path}.argv must contain 1 to 100 arguments.`); + command.argv.forEach((argument, argumentIndex) => requiredString(argument, `${path}.argv[${argumentIndex}]`)); + validateRelativePath(command.cwd, `${path}.cwd`, { allowDot: true }); + if (!Number.isInteger(command.timeoutMs) || command.timeoutMs < 100 || command.timeoutMs > 60 * 60 * 1000) { + throw new Error(`${path}.timeoutMs must be between 100 and 3600000.`); + } + boolean(command.required, `${path}.required`); + } + return value; +} + +export function validateValidationResult(value) { + object(value, "validation result"); + exactKeys(value, ["schemaVersion", "runId", "repository", "revision", "suite", "runner", "status", "checkpoints", "commands", "startedAt", "completedAt", "integrity"], "validation result"); + equals(value.schemaVersion, VALIDATION_RESULT_SCHEMA_VERSION, "validation result.schemaVersion"); + requiredString(value.runId, "validation result.runId"); + object(value.repository, "validation result.repository"); + exactKeys(value.repository, ["id"], "validation result.repository"); + requiredString(value.repository.id, "validation result.repository.id"); + object(value.revision, "validation result.revision"); + exactKeys(value.revision, ["baseCommit", "mergeBase", "headCommit"], "validation result.revision"); + oid(value.revision.baseCommit, "validation result.revision.baseCommit"); + oid(value.revision.mergeBase, "validation result.revision.mergeBase"); + oid(value.revision.headCommit, "validation result.revision.headCommit"); + object(value.suite, "validation result.suite"); + exactKeys(value.suite, ["id", "manifestPath", "manifestDigest"], "validation result.suite"); + requiredString(value.suite.id, "validation result.suite.id"); + validateRelativePath(value.suite.manifestPath, "validation result.suite.manifestPath"); + sha256(value.suite.manifestDigest, "validation result.suite.manifestDigest"); + object(value.runner, "validation result.runner"); + exactKeys(value.runner, ["id", "runtime"], "validation result.runner"); + requiredString(value.runner.id, "validation result.runner.id"); + requiredString(value.runner.runtime, "validation result.runner.runtime"); + member(value.status, ["passed", "failed"], "validation result.status"); + stringArray(value.checkpoints, "validation result.checkpoints"); + if (!Array.isArray(value.commands) || value.commands.length === 0) throw new Error("validation result.commands must be a non-empty array."); + value.commands.forEach((command, index) => validateCommandResult(command, `validation result.commands[${index}]`)); + date(value.startedAt, "validation result.startedAt"); + date(value.completedAt, "validation result.completedAt"); + object(value.integrity, "validation result.integrity"); + exactKeys(value.integrity, ["algorithm", "digest"], "validation result.integrity"); + equals(value.integrity.algorithm, "sha256", "validation result.integrity.algorithm"); + sha256(value.integrity.digest, "validation result.integrity.digest"); + if (validationResultDigest(value) !== value.integrity.digest) throw new Error("validation result integrity digest does not match."); + const expectedStatus = value.commands.some((command) => command.required && command.status !== "passed") ? "failed" : "passed"; + if (value.status !== expectedStatus) throw new Error("validation result status does not match required command results."); + return value; +} + +async function runValidationCommand(command, workspace, home) { + const cwd = resolve(workspace, command.cwd); + if (relative(workspace, cwd).startsWith("..") || isAbsolute(relative(workspace, cwd))) throw new Error(`Command ${command.id} cwd escapes validation workspace.`); + const startedAt = new Date(); + const execution = await spawnCaptured(command.argv, { + cwd, + timeoutMs: command.timeoutMs, + home, + }); + const completedAt = new Date(); + return { + id: command.id, + argv: [...command.argv], + cwd: command.cwd, + required: command.required, + status: execution.timedOut ? "timed_out" : execution.spawnError ? "error" : execution.exitCode === 0 ? "passed" : "failed", + exitCode: execution.exitCode, + signal: execution.signal, + durationMs: completedAt.getTime() - startedAt.getTime(), + stdout: execution.stdout, + stderr: execution.stderr, + startedAt: startedAt.toISOString(), + completedAt: completedAt.toISOString(), + error: execution.spawnError, + }; +} + +function skippedCommand(command) { + return { + id: command.id, + argv: [...command.argv], + cwd: command.cwd, + required: command.required, + status: "skipped", + exitCode: null, + signal: null, + durationMs: 0, + stdout: emptyOutput(), + stderr: emptyOutput(), + startedAt: null, + completedAt: null, + error: null, + }; +} + +function spawnCaptured(argv, { cwd, timeoutMs, home }) { + return new Promise((resolvePromise) => { + const child = spawn(argv[0], argv.slice(1), { + cwd, + env: validationEnvironment(home), + stdio: ["ignore", "pipe", "pipe"], + }); + const stdout = outputAccumulator(); + const stderr = outputAccumulator(); + let spawnError = null; + let timedOut = false; + let killTimer = null; + const timer = setTimeout(() => { + timedOut = true; + child.kill("SIGTERM"); + killTimer = setTimeout(() => child.kill("SIGKILL"), 1_000); + }, timeoutMs); + child.stdout?.on("data", stdout.add); + child.stderr?.on("data", stderr.add); + child.once("error", (error) => { spawnError = error.message; }); + child.once("close", (exitCode, signal) => { + clearTimeout(timer); + if (killTimer) clearTimeout(killTimer); + resolvePromise({ + exitCode: Number.isInteger(exitCode) ? exitCode : null, + signal: signal ?? null, + timedOut, + spawnError, + stdout: stdout.finish(), + stderr: stderr.finish(), + }); + }); + }); +} + +function outputAccumulator() { + const hash = createHash("sha256"); + let bytes = 0; + let tail = Buffer.alloc(0); + return { + add(chunk) { + const buffer = Buffer.from(chunk); + bytes += buffer.length; + hash.update(buffer); + tail = Buffer.concat([tail, buffer]).subarray(-MAX_OUTPUT_TAIL_BYTES); + }, + finish() { + return { + bytes, + digest: hash.digest("hex"), + tail: tail.toString("utf8"), + truncated: bytes > MAX_OUTPUT_TAIL_BYTES, + }; + }, + }; +} + +function emptyOutput() { + return { bytes: 0, digest: createHash("sha256").update("").digest("hex"), tail: "", truncated: false }; +} + +async function checkpointIds(cwd, baseCommit, headCommit) { + const result = await runGit({ + args: ["log", "--format=%(trailers:key=Entire-Checkpoint,valueonly)", "--no-merges", `${baseCommit}..${headCommit}`], + cwd, + }); + return [...new Set(result.stdout.split(/\r?\n/).map((value) => value.trim()).filter(Boolean))].sort(); +} + +async function writeResultWithRetry(ledger, path, result) { + for (let attempt = 0; attempt < 3; attempt += 1) { + const current = await ledger.read(path); + if (current.value !== null) throw new Error(`Validation result already exists: ${path}.`); + try { + return await ledger.write(path, result, { expectedVersion: current.version }); + } catch (error) { + if (!(error instanceof LedgerConflictError) || attempt === 2) throw error; + } + } + throw new Error("Unable to write validation result after concurrent updates."); +} + +function validationPath(commit, runId) { + return `commits/${commit}/${runId}.json`; +} + +function validationResultDigest(value) { + const { integrity: _integrity, ...unsigned } = value; + return digestObject(unsigned); +} + +function validationEnvironment(home) { + const allowed = ["PATH", "LANG", "LC_ALL", "TERM"]; + const env = Object.fromEntries(allowed.flatMap((key) => process.env[key] === undefined ? [] : [[key, process.env[key]]])); + return { ...env, HOME: home, TMPDIR: resolve(home, "tmp"), CI: "1", NO_COLOR: "1", GIT_TERMINAL_PROMPT: "0" }; +} + +function validateCommandResult(value, path) { + object(value, path); + exactKeys(value, ["id", "argv", "cwd", "required", "status", "exitCode", "signal", "durationMs", "stdout", "stderr", "startedAt", "completedAt", "error"], path); + requiredString(value.id, `${path}.id`); + if (!Array.isArray(value.argv) || value.argv.length === 0) throw new Error(`${path}.argv must be non-empty.`); + value.argv.forEach((argument, index) => requiredString(argument, `${path}.argv[${index}]`)); + validateRelativePath(value.cwd, `${path}.cwd`, { allowDot: true }); + boolean(value.required, `${path}.required`); + member(value.status, ["passed", "failed", "error", "timed_out", "skipped"], `${path}.status`); + if (value.exitCode !== null && !Number.isInteger(value.exitCode)) throw new Error(`${path}.exitCode must be an integer or null.`); + if (value.signal !== null) requiredString(value.signal, `${path}.signal`); + if (!Number.isInteger(value.durationMs) || value.durationMs < 0) throw new Error(`${path}.durationMs must be a non-negative integer.`); + validateOutput(value.stdout, `${path}.stdout`); + validateOutput(value.stderr, `${path}.stderr`); + if (value.startedAt !== null) date(value.startedAt, `${path}.startedAt`); + if (value.completedAt !== null) date(value.completedAt, `${path}.completedAt`); + if (value.error !== null) requiredString(value.error, `${path}.error`); + if (value.status === "skipped" && (value.startedAt !== null || value.completedAt !== null)) throw new Error(`${path} skipped timestamps must be null.`); +} + +function validateOutput(value, path) { + object(value, path); + exactKeys(value, ["bytes", "digest", "tail", "truncated"], path); + if (!Number.isInteger(value.bytes) || value.bytes < 0) throw new Error(`${path}.bytes must be a non-negative integer.`); + sha256(value.digest, `${path}.digest`); + if (typeof value.tail !== "string") throw new Error(`${path}.tail must be a string.`); + boolean(value.truncated, `${path}.truncated`); +} + +function exactKeys(value, expected, path) { + const actual = Object.keys(value).sort(); + const wanted = [...expected].sort(); + if (JSON.stringify(actual) !== JSON.stringify(wanted)) throw new Error(`${path} must contain exactly: ${wanted.join(", ")}.`); +} + +function object(value, path) { + if (typeof value !== "object" || value === null || Array.isArray(value)) throw new Error(`${path} must be an object.`); +} + +function requiredString(value, path) { + if (typeof value !== "string" || value === "" || /[\0\r\n]/.test(value)) throw new Error(`${path} must be a non-empty single-line string.`); +} + +function boolean(value, path) { + if (typeof value !== "boolean") throw new Error(`${path} must be a boolean.`); +} + +function member(value, values, path) { + if (!values.includes(value)) throw new Error(`${path} must be one of: ${values.join(", ")}.`); +} + +function equals(value, expected, path) { + if (value !== expected) throw new Error(`${path} must be ${JSON.stringify(expected)}.`); +} + +function oid(value, path) { + if (typeof value !== "string" || !/^(?:[0-9a-f]{40}|[0-9a-f]{64})$/.test(value)) throw new Error(`${path} must be a Git object ID.`); +} + +function sha256(value, path) { + if (typeof value !== "string" || !/^[0-9a-f]{64}$/.test(value)) throw new Error(`${path} must be a SHA-256 digest.`); +} + +function date(value, path) { + requiredString(value, path); + if (Number.isNaN(Date.parse(value))) throw new Error(`${path} must be an ISO date-time.`); +} + +function stringArray(value, path) { + if (!Array.isArray(value) || value.some((item) => typeof item !== "string" || item === "")) throw new Error(`${path} must be an array of non-empty strings.`); + if (new Set(value).size !== value.length) throw new Error(`${path} must contain unique values.`); +} + +function validateRelativePath(value, path, { allowDot = false } = {}) { + requiredString(value, path); + if ((allowDot && value === ".")) return; + if (value.startsWith("/") || value.includes("\\") || value.split("/").some((segment) => segment === "" || segment === "." || segment === "..")) { + throw new Error(`${path} must be a safe relative path.`); + } +} diff --git a/scripts/tabellio-review.mjs b/scripts/tabellio-review.mjs index fec66fb..a090873 100755 --- a/scripts/tabellio-review.mjs +++ b/scripts/tabellio-review.mjs @@ -17,6 +17,10 @@ try { repoPath: store.repoPath, ref: options.ledgerRef ?? "refs/tabellio/reviews", }); + const validationLedger = await GitJsonLedger.open({ + repoPath: store.repoPath, + ref: options.validationLedgerRef ?? "refs/tabellio/validations", + }); let provider = null; if (options.command === "sync") { const token = (await readFile(resolve(options.tokenFile), "utf8")).trim(); @@ -26,6 +30,7 @@ try { const manager = new ReviewCycleManager({ store, ledger, + validationLedger, provider, repositoryId, owner: options.owner, @@ -84,7 +89,7 @@ function parseArgs(args) { if (Object.hasOwn(values, key)) throw new Error(`Duplicate option: ${flag}.`); values[key] = value; } - const common = ["repo", "repoId", "owner", "forgeRepo", "number", "actor", "ledgerRef"]; + const common = ["repo", "repoId", "owner", "forgeRepo", "number", "actor", "ledgerRef", "validationLedgerRef"]; const allowed = { sync: [...common, "baseUrl", "tokenFile"], status: common, diff --git a/scripts/tabellio-validate.mjs b/scripts/tabellio-validate.mjs new file mode 100755 index 0000000..3c8e0f1 --- /dev/null +++ b/scripts/tabellio-validate.mjs @@ -0,0 +1,59 @@ +#!/usr/bin/env node + +import { resolve } from "node:path"; + +import { GitJsonLedger } from "./lib/git-json-ledger.mjs"; +import { repositoryIdentity } from "./lib/repository-identity.mjs"; +import { latestValidationResult, ValidationRunner } from "./lib/validation-runner.mjs"; +import { NativeGitStore } from "./providers/native-git-store.mjs"; + +try { + const options = parseArgs(process.argv.slice(2)); + const store = await NativeGitStore.open(resolve(options.repo ?? process.cwd())); + const ledger = await GitJsonLedger.open({ + repoPath: store.repoPath, + ref: options.ledgerRef ?? "refs/tabellio/validations", + }); + if (options.command === "run") { + const runner = new ValidationRunner({ store, ledger }); + const result = await runner.run({ + repositoryId: await repositoryIdentity(store, options.repoId), + commit: options.commit ?? "HEAD", + base: options.base ?? "main", + manifestPath: options.manifest ?? "tabellio.validation.json", + runnerId: options.runnerId ?? "local", + }); + console.log(JSON.stringify({ ok: true, ...result }, null, 2)); + } else { + const commit = await store.resolveRef(options.commit ?? "HEAD"); + const result = await latestValidationResult(ledger, commit); + console.log(JSON.stringify({ ok: result !== null, commit, result }, null, 2)); + if (result === null) process.exitCode = 1; + } +} catch (error) { + process.exitCode = 1; + console.error(JSON.stringify({ + ok: false, + error: error instanceof Error ? error.message : String(error), + name: error instanceof Error ? error.name : "Error", + }, null, 2)); +} + +function parseArgs(args) { + const command = args[0]; + if (!["run", "latest"].includes(command)) throw new Error("Expected command: run or latest."); + const values = {}; + for (let index = 1; index < args.length; index += 2) { + const flag = args[index]; + const value = args[index + 1]; + if (!flag?.startsWith("--") || value === undefined) throw new Error(`Expected a value after ${flag ?? command}.`); + const key = flag.slice(2).replace(/-([a-z])/g, (_match, letter) => letter.toUpperCase()); + if (Object.hasOwn(values, key)) throw new Error(`Duplicate option: ${flag}.`); + values[key] = value; + } + const allowed = command === "run" + ? ["repo", "repoId", "commit", "base", "manifest", "runnerId", "ledgerRef"] + : ["repo", "commit", "ledgerRef"]; + for (const key of Object.keys(values)) if (!allowed.includes(key)) throw new Error(`Unsupported option: --${key}.`); + return { command, ...values }; +} diff --git a/tabellio.validation.json b/tabellio.validation.json new file mode 100644 index 0000000..f1c366d --- /dev/null +++ b/tabellio.validation.json @@ -0,0 +1,15 @@ +{ + "schemaVersion": "tabellio-validation/v0.1", + "id": "tabellio-default", + "failFast": true, + "requireEntireCheckpoint": true, + "commands": [ + { + "id": "repository-check", + "argv": ["npm", "run", "check"], + "cwd": ".", + "timeoutMs": 1200000, + "required": true + } + ] +} diff --git a/tests/review-cycle.test.mjs b/tests/review-cycle.test.mjs index 43bd35e..501750f 100644 --- a/tests/review-cycle.test.mjs +++ b/tests/review-cycle.test.mjs @@ -8,6 +8,7 @@ import { validateReviewCycle, } from "../scripts/lib/review-cycle.mjs"; import { runGit } from "../scripts/lib/git-process.mjs"; +import { digestObject } from "../scripts/lib/stack-operation.mjs"; import { NativeGitStore } from "../scripts/providers/native-git-store.mjs"; import { createFixture, identityEnv } from "./helpers/git-fixture.mjs"; @@ -145,6 +146,39 @@ test("review cycle persists forge and agent feedback through triage and checkpoi assert.equal(worktree.stdout, ""); }); +test("review readiness consumes only the latest validation for the exact PR head", async (t) => { + const fixture = await createFixture(); + t.after(() => rm(fixture.root, { recursive: true, force: true })); + const store = await NativeGitStore.open(fixture.seed); + const ledger = await GitJsonLedger.open({ repoPath: fixture.seed, ref: "refs/tabellio/reviews" }); + const validationLedger = await GitJsonLedger.open({ repoPath: fixture.seed, ref: "refs/tabellio/validations" }); + const provider = emptyProvider(fixture); + const manager = new ReviewCycleManager({ + store, + ledger, + validationLedger, + provider, + repositoryId: "example/repository", + owner: "acme", + repo: "project", + }); + + let result = await manager.sync({ number: 7, actor: "sync-agent", now: new Date(timestamp) }); + assert.equal(result.cycle.status, "validating"); + const passed = validationResult(fixture.featureCommit, "validation-pass", "passed", "2026-07-10T20:01:00.000Z"); + let current = await validationLedger.read(`commits/${fixture.featureCommit}/${passed.runId}.json`); + await validationLedger.write(`commits/${fixture.featureCommit}/${passed.runId}.json`, passed, { expectedVersion: current.version }); + result = await manager.sync({ number: 7, actor: "sync-agent", now: new Date("2026-07-10T20:02:00.000Z") }); + assert.equal(result.cycle.status, "ready"); + assert.equal(result.cycle.checks.statuses[0].context, "tabellio/test-suite"); + + const failed = validationResult(fixture.featureCommit, "validation-fail", "failed", "2026-07-10T20:03:00.000Z"); + current = await validationLedger.read(`commits/${fixture.featureCommit}/${failed.runId}.json`); + await validationLedger.write(`commits/${fixture.featureCommit}/${failed.runId}.json`, failed, { expectedVersion: current.version }); + result = await manager.sync({ number: 7, actor: "sync-agent", now: new Date("2026-07-10T20:04:00.000Z") }); + assert.equal(result.cycle.status, "blocked"); +}); + function fakeProvider(fixture) { let checkState = "failure"; let headCommit = fixture.featureCommit; @@ -237,3 +271,63 @@ function fakeProvider(fixture) { }, }; } + +function emptyProvider(fixture) { + return { + async changeRequest() { + return { + id: "21", + number: 7, + title: "Agent change", + state: "open", + draft: false, + mergeable: true, + source: { branch: "feature", commit: fixture.featureCommit }, + target: { branch: "main", commit: fixture.mainCommit }, + author: "agent", + webUrl: "https://forgejo.example.test/acme/project/pulls/7", + createdAt: timestamp, + updatedAt: timestamp, + }; + }, + async listReviews() { return []; }, + async listReviewComments() { return []; }, + async listIssueComments() { return []; }, + async commitStatus() { + return { commit: fixture.featureCommit, state: "none", total: 0, statuses: [] }; + }, + }; +} + +function validationResult(commit, runId, status, completedAt) { + const commandStatus = status === "passed" ? "passed" : "failed"; + const value = { + schemaVersion: "tabellio-validation-result/v0.1", + runId, + repository: { id: "example/repository" }, + revision: { baseCommit: "a".repeat(40), mergeBase: "a".repeat(40), headCommit: commit }, + suite: { id: "test-suite", manifestPath: "tabellio.validation.json", manifestDigest: "c".repeat(64) }, + runner: { id: "test", runtime: "node-test" }, + status, + checkpoints: ["checkpoint-001"], + commands: [{ + id: "tests", + argv: ["npm", "test"], + cwd: ".", + required: true, + status: commandStatus, + exitCode: status === "passed" ? 0 : 1, + signal: null, + durationMs: 1, + stdout: { bytes: 0, digest: "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", tail: "", truncated: false }, + stderr: { bytes: 0, digest: "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", tail: "", truncated: false }, + startedAt: "2026-07-10T20:00:00.000Z", + completedAt, + error: null, + }], + startedAt: "2026-07-10T20:00:00.000Z", + completedAt, + }; + value.integrity = { algorithm: "sha256", digest: digestObject(value) }; + return value; +} diff --git a/tests/validation-runner.test.mjs b/tests/validation-runner.test.mjs new file mode 100644 index 0000000..6ac8585 --- /dev/null +++ b/tests/validation-runner.test.mjs @@ -0,0 +1,124 @@ +import assert from "node:assert/strict"; +import { rm, writeFile } from "node:fs/promises"; +import test from "node:test"; + +import { GitJsonLedger } from "../scripts/lib/git-json-ledger.mjs"; +import { runGit } from "../scripts/lib/git-process.mjs"; +import { repositoryIdentity } from "../scripts/lib/repository-identity.mjs"; +import { + latestValidationResult, + ValidationRunner, + validateValidationManifest, + validateValidationResult, +} from "../scripts/lib/validation-runner.mjs"; +import { NativeGitStore } from "../scripts/providers/native-git-store.mjs"; +import { createFixture, identityEnv } from "./helpers/git-fixture.mjs"; + +test("validation runner executes exact committed manifests and stores bounded results", async (t) => { + const fixture = await createFixture(); + t.after(() => rm(fixture.root, { recursive: true, force: true })); + await runGit({ args: ["switch", "feature"], cwd: fixture.seed }); + const manifestPath = `${fixture.seed}/tabellio.validation.json`; + await writeFile(manifestPath, JSON.stringify(manifest([ + command("tests", [process.execPath, "-e", 'process.stdout.write("x".repeat(20000))']), + command("isolated-home", [process.execPath, "-e", 'if (!process.env.HOME.includes("validation-workspaces")) process.exit(2)']), + ]), null, 2)); + await commit(fixture.seed, "Add passing validation", "validation-pass"); + const passingHead = await head(fixture.seed); + + const store = await NativeGitStore.open(fixture.seed); + const ledger = await GitJsonLedger.open({ repoPath: fixture.seed, ref: "refs/tabellio/validations" }); + const runner = new ValidationRunner({ store, ledger }); + const repositoryId = await repositoryIdentity(store, "example/repository"); + const passed = await runner.run({ + repositoryId, + commit: passingHead, + base: "main", + runnerId: "test-runner", + now: new Date("2026-07-10T20:00:00.000Z"), + }); + assert.equal(passed.result.status, "passed"); + assert.deepEqual(passed.result.checkpoints, ["validation-pass"]); + assert.equal(passed.result.commands[0].stdout.bytes, 20000); + assert.equal(passed.result.commands[0].stdout.truncated, true); + assert.equal(Buffer.byteLength(passed.result.commands[0].stdout.tail), 16 * 1024); + assert.equal(passed.result.commands[1].status, "passed"); + assert.equal(validateValidationResult(passed.result), passed.result); + assert.deepEqual(await latestValidationResult(ledger, passingHead), passed.result); + + await writeFile(manifestPath, JSON.stringify(manifest([ + command("fails", [process.execPath, "-e", "process.exit(3)"]), + command("skipped", [process.execPath, "-e", "process.exit(0)"]), + ]), null, 2)); + await commit(fixture.seed, "Add failing validation", "validation-fail"); + const failingHead = await head(fixture.seed); + const failed = await runner.run({ + repositoryId, + commit: failingHead, + base: "main", + runnerId: "test-runner", + }); + assert.equal(failed.result.status, "failed"); + assert.equal(failed.result.commands[0].exitCode, 3); + assert.equal(failed.result.commands[1].status, "skipped"); + assert.deepEqual(failed.result.checkpoints, ["validation-fail", "validation-pass"]); + + const worktrees = await runGit({ args: ["worktree", "list", "--porcelain"], cwd: fixture.seed }); + assert.equal(worktrees.stdout.includes(passed.result.runId), false); + const status = await runGit({ args: ["status", "--porcelain=v1"], cwd: fixture.seed }); + assert.equal(status.stdout, ""); +}); + +test("validation manifest rejects shell-like ambiguity and missing checkpoint ranges", async (t) => { + assert.throws( + () => validateValidationManifest(manifest([command("escape", ["node", "test.js"], "../outside")])), + /safe relative path/, + ); + assert.throws( + () => validateValidationManifest({ ...manifest([]), commands: [] }), + /1 to 50/, + ); + + const fixture = await createFixture(); + t.after(() => rm(fixture.root, { recursive: true, force: true })); + await runGit({ args: ["switch", "feature"], cwd: fixture.seed }); + await writeFile(`${fixture.seed}/tabellio.validation.json`, JSON.stringify(manifest([ + command("tests", [process.execPath, "-e", "process.exit(0)"]), + ]), null, 2)); + await runGit({ args: ["add", "tabellio.validation.json"], cwd: fixture.seed }); + await runGit({ args: ["commit", "-m", "Manifest without checkpoint"], cwd: fixture.seed, env: identityEnv() }); + const store = await NativeGitStore.open(fixture.seed); + const ledger = await GitJsonLedger.open({ repoPath: fixture.seed, ref: "refs/tabellio/validations" }); + const runner = new ValidationRunner({ store, ledger }); + await assert.rejects( + runner.run({ repositoryId: "example/repository", commit: "HEAD", base: "main" }), + /has no Entire checkpoint/, + ); +}); + +function manifest(commands) { + return { + schemaVersion: "tabellio-validation/v0.1", + id: "test-suite", + failFast: true, + requireEntireCheckpoint: true, + commands, + }; +} + +function command(id, argv, cwd = ".") { + return { id, argv, cwd, timeoutMs: 10_000, required: true }; +} + +async function commit(cwd, message, checkpoint) { + await runGit({ args: ["add", "tabellio.validation.json"], cwd }); + await runGit({ + args: ["commit", "-m", message, "-m", `Entire-Checkpoint: ${checkpoint}`], + cwd, + env: identityEnv(), + }); +} + +async function head(cwd) { + return (await runGit({ args: ["rev-parse", "HEAD"], cwd })).stdout.trim(); +}