Skip to content

Phase 1 / Etch / Test-runner + test identity#44

Merged
guysenpai merged 20 commits into
mainfrom
phase-1/etch/test-runner
Jul 5, 2026
Merged

Phase 1 / Etch / Test-runner + test identity#44
guysenpai merged 20 commits into
mainfrom
phase-1/etch/test-runner

Conversation

@guysenpai

Copy link
Copy Markdown
Contributor

Milestone M1.0.15 — Etch test-runner + test identity. Brief: briefs/M1.0.15-test-runner.md.

test "X" { } blocks now execute end-to-end (parse + validate + register existed since M1.0.8; zero execution surface until now), closing the M1.0.15 line of the C1.6 gap plan. Delivered gate-by-gate E1→E5.

Closing notes

What worked

  • The E1 recon's five points held exactly — compile() is the full type-registration point, so "fresh World + Interpreter per test" fell out cleanly; no gap in bindToWorld.
  • Reusing existing primitives paid off: spawn_with shares the observer-firing spawn path (spawnWithObservers factored from the deferred .spawn flush); emit method shares enqueueEvent; assert string comparison reuses eventValueEql; E0101 reused for duplicate tests (no new code).
  • Adversarial-review workflows earned their cost: caught 5 real E4 defects (incl. a dangerous assert_neq false-PASS on strings) before they shipped.
  • Break-tests (neuter a fix/assertion, confirm the failure, revert) proved both the UAF fix and that the new inline/driver tests actually execute — after the §13 wire-in gap surfaced.

What deviated from the original spec / brief plan

  • Two mechanisms not in the brief's file-level plan, both minimal + confined and ack'd by STOP fixes: suppress_body_store_resets (E3 UAF) and in_test_body runtime flag (E4 test-scoped-builtin gate).
  • A Tier-0 touch (observers.zig/world.zig spawnWithObservers) required by the brief's own "reuse the same application primitive" trap — one spawn path, no flush behavior change.
  • test-etch runs the shim over the green fixtures in ONE invocation (clean un-interleaved ✓ output) rather than one Run step per fixture.

What to flag explicitly in review

  • Event-clear semantics for emit; tick(1): stepOnce clears the event store at its head, so tick/tick_until set a one-shot suppress_event_clear to let a pre-tick event reach the first tick (§32). Non-test paths never set it.
  • assert_eq/assert_neq are intentionally MORE capable than == (which rejects strings at type-check): they compare strings by bytes and reject aggregates — a test-only Eq surface, so assert_neq cannot silently false-PASS.
  • The hunt:semantics adversarial agent stalled (never returned); the other three lenses + inline/driver tests cover measure/tick_until/Duration/asserts.

Final measurements — no perf gate this milestone. zig build test green debug + ReleaseSafe; zig build test-etch prints total: 10 passed / 0 failed / 2 skipped (exit 0); measure around a 5000-iter interp loop reports ~5 ms wall-clock. New diagnostic: E0910 (sole new code).

Residual risks / tech debt (intentional)

  • Memory during a long tick(n)/tick_until inside a test grows (per-body store resets suppressed) — bounded by the test body's end (freed en bloc); test-scale, acked at the E3 STOP.
  • test_world() is mono-world (spec'd v0.6 restriction, not debt); multi-world, async test bodies, the weld CLI proper, and filtering flags are out per §32.

Validation (Step 4)

  • All "Scope" deliverables present (E1 identity + annotations + body type-check; E2 runner library; E3 world surface; E4 assertions + measure + tick_until; E5 shim + corpus + CI).
  • No drift into "Out of scope".
  • All "Acceptance criteria › Tests" pass in debug and ReleaseSafe (driver_test.zig + inline suites).
  • "Observable behavior" demonstrable — zig build test-etch prints ✓/skip lines + aggregate (exit 0); etch_test failing.etch prints the ✗ line + file:line + exit 1.
  • zig build, zig build test, zig build test-etch, zig fmt --check, zig build lint all green.
  • CLAUDE.md updated (§3.4 current-state table + Tags row + open-decisions bullet) — commit docs(claude-md): update for M1.0.15.
  • Language audit (§3.6.1) — no FR function-words in the branch diff or brief (the lone .le match is the Zig <= enum tag).
  • Local drift audit (§3.6.1) — stale kw_test comment refreshed; no residual SymbolKind.test_ (removed E2); .test_ matches are AnnotTarget.test_.
  • Brief "Closing notes" filled; Status: CLOSED, Closed: 2026-07-05; final commit docs(brief): close M1.0.15.

Merge + tag (v0.10.15-test-runner) + close decision are Guy's.

🤖 Generated with Claude Code

guysenpai and others added 20 commits July 5, 2026 13:12
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
M1.0.15 E1. Separate `test_symbols` namespace so `test "Foo"` no longer
collides with `component Foo` (E0101, contextual "duplicate test 'X'" for
intra-namespace dups). `TestDecl` keeps its `@tag`/`@skip`/`@only`
annotation range (was discarded); applicability rides `validateAnnotations`
with a new `.test_` target, args validated (reusing E0502, no new code).
Test bodies are type-checked as a sync context (`await` -> E0901); the
`in_test_body` flag gates the test-scoped builtins landing in E3/E4.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
M1.0.15 E2. `test_runner.zig`: iterate a type-checked program's `test`
blocks in declaration order, each in a fresh World + Interpreter (full
isolation), honouring `@skip`/`@only`, into a `RunReport` (per-test
name/status/duration/message/span + passed/failed/skipped aggregate; owns
its strings). `interp.runTestBody` runs one body sync-to-completion and
converts a failed assert / uncaught throw / runtime failure into a
`.fail` outcome (span + message), not an aggregated count. The `assert`
statement now raises `fail(.AssertFailed, span)` with its literal message
(`RuntimeErrorKind.AssertFailed`, `pending_message`, owned `test_msg_buf`
for E4 formatted messages). Runner + report types exported from root.zig.

Absorbs the E1-review nit: drop the now-dead `SymbolKind.test_` variant
and fix the stale buildExports comment.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
M1.0.15 E3. `Value.world_handle` + `ResolvedType.test_world`: `test_world()`
(test-body-gated builtin) returns the mono-world handle; `spawn_with([C {…}])`
spawns immediately and returns a live Entity (shares the observer-firing
primitive `world.spawnWithObservers`, factored out of applyWithObservers's
.spawn arm), `emit(T {…})` mirrors the emit statement (shared enqueueEvent),
`tick(n)` advances n ticks. `world.emit(...)` parses via a new `.kw_emit`
postfix arm (emit is a keyword). `stepOnce`'s head event-clear is suppressed
for a test's first tick so `emit; tick(1)` delivers the event (§32); the store
is cleared at the end of tick(). Test bodies get unrestricted ECS access.

Also fixes two latent defects the E3 test wire-in exposed: test_runner.zig's
inline tests were never collected (missing from root.zig's comptime import
guard, §13), and `std.time.Timer` does not exist in Zig 0.16 — per-test
duration now uses `std.Io.Clock` with a threaded `io`.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
M1.0.15 E3 review fix. A `test` body outlives the rule/guard/timer/observer/
hook bodies that `world.tick(n)` drives; those bodies' per-body resets of the
shared value stores (collections/closures/structs/optionals/run-strings) freed
heap-backed values the test's locals still held — a use-after-free for a
`.string_run` (concat) held across a tick (the M1.0.14 class), and stale/OOB
handles for collections/structs/closures. Route every per-body reset through a
single `resetBodyStores()` gated by `suppress_body_store_resets`, which
`runTestBody` holds for the whole body: the driven bodies accumulate into the
shared stores (test-scale, bounded) and the test's own raw end-defers free
everything at once. No production-path change (flag inactive outside a test).
Two regression tests (collection + string_run held across tick(1)) abort under
testing.allocator with the guard neutered, pass with it.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
M1.0.15 E4. Adds the §19.4 assertion builtins (assert_eq/assert_neq/
assert_approx (tol 1e-6)/assert_some/assert_none) + panic/todo/unreachable
(global) and the test-scoped tick_until(pred: () -> bool, timeout) -> bool,
dispatched via synthBuiltinCall (types) + evalBuiltinCall (interp) which folds
the E3 test_world. The `measure { block }` expression (new kw_measure token,
ExprKind.measure_expr, E0910 MeasureOutsideTest, interp eval via std.Io.Clock)
returns wall-clock Duration, test-body only. Duration ordering added to
binaryCompare (the checker already typed it). Assert failures format both
values into test_msg_buf.

Includes 5 fixes from an adversarial review of this diff: harvestError clears
the stale pending_message; test_world/tick_until are gated by a runtime
in_test_body flag (checker/interp parity); assert_eq/neq compare strings by
bytes (eventValueEql) and reject non-comparable aggregate operands at
type-check; a throwing tick_until predicate stops the loop and surfaces.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
M1.0.15 E4 review fix. assert_approx only synth'd its args, so
assert_approx(1, 2) type-checked clean then failed loud at runtime —
asymmetric with the rest of the assert family, which is fail-loud at
type-check. synthBuiltinCall now requires a, b, and the optional tolerance
to be float (isFloatType), emitting type_mismatch otherwise; the interp
float check stays as a defensive belt.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
M1.0.15 E5. tools/etch_test — a thin CLI shim over weld_etch.test_runner
(etch_cook pattern: arg parsing + I/O + report printing only): parse ->
type-check -> run each .etch file, print per-test ✓/✗/skipped lines with
durations + aggregates, clean nonzero exit on any diagnostic/failure. build.zig
adds the etch_test exe + a `test-etch` step (Zig acceptance driver + the shim
over the green fixtures in one invocation); the driver also joins `zig build
test`, and CI runs `zig build test-etch` per leg. Acceptance corpus
tests/etch/test_runner/: .etch fixtures (green/failing/only/isolation/world/
timing) embedded + inline diagnostic cases in driver_test.zig — the 13
acceptance tests under testing.allocator.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
§3.6.1 drift audit: the kw_test token comment still said "parse + validate
only, no execution" — tests now run end-to-end via the M1.0.15 runner.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@guysenpai guysenpai merged commit 32b1188 into main Jul 5, 2026
10 checks passed
@guysenpai guysenpai deleted the phase-1/etch/test-runner branch July 5, 2026 21:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant