Ixon: .ixe bundle format — main/assumptions header, metadata-last sections, blob integrity#474
Open
johnchandlerburnham wants to merge 1 commit into
Open
Ixon: .ixe bundle format — main/assumptions header, metadata-last sections, blob integrity#474johnchandlerburnham wants to merge 1 commit into
johnchandlerburnham wants to merge 1 commit into
Conversation
…tions, blob integrity Evolves .ixe in place (Tag4(0xE,0) kept, pre-alpha) so a single env file can pin one Lean value as a self-contained bundle: - Header gains main : Option Address (bundle root; writers and all readers enforce main ∈ consts) and a strictly-ascending assumptions list (thin-bundle trust boundary; merkle_root_canonical over the leaves reproduces Claim.assumptions roots). - Sections reordered hot-first: blobs, consts, anon_hints, names, named, comms. get_anon/get_anon_mmap stop after §3 (no metadata parse-and-discard); parse_lazy_index stops after §5; the full readers reject trailing bytes. - anon_hints promoted from optional trailer to an always-written §3, derived from Named Def metadata when the map is empty; read-time hint harvesting removed (EnvHandle::from_bytes post-pass dropped). - Blob entries are hash-verified in every reader on both sides (a swapped blob silently changed a Nat/String literal before); Lean getEnv also gains the const hash check it was missing. - New Env::bfs_closure follows all three edge kinds (Constant.refs, Prj→block, Muts→member/ctor projections — a refs-only walk returns just the root for projections); prune_to_closure builds bundles (value closure + named-metadata pruning: name parent chains and string blobs, DataValue payload blobs, meta_refs edges, aux_gen originals, per-const hints); validate_closed is the receiver check. kernel::anon_work::closure_addrs now delegates to bfs_closure. - Lean topologicalSortNames emits the anonymous name at index 0, matching Rust: serEnv and rsSerEnv are now byte-identical (verified byte-exact on a 505 MB whole-stdlib env) and the putIdx getD-0 fallback now resolves to anon instead of colliding with the first real name. - FFI mirrors: RawEnv carries main/assumptions/anonHints (ctor arity 5→8), RawEnvLazy carries main/assumptions (3→5); the manual RawEnv builders in ffi compile.rs populate the new slots via set_raw_env_bundle_fields. - Generators/fixtures content-address blobs and exercise the bundle fields; tampering tests compute offsets programmatically instead of hardcoding the header layout; docs/Ixon.md spec updated.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Merge #473 into main first.
Evolves the
.ixeformat in place (pre-alpha, Tag4(0xE,0) kept, no compat shims) so asingle env file can pin one Lean value as a self-contained, verifiable bundle — the
groundwork for using Ixon as a client/server interchange format. A constant's address
already pins its entire dependency DAG (refs tables are content addresses, recursively);
this PR adds the transport half: a distinguished root, an explicit trust boundary for
thin bundles, and closure helpers to produce/validate them.
Wire format
get_anon/get_anon_mmapread §1–§3 and stop — no moreparse-and-discard of names/named/comms;
parse_lazy_indexstops before comms; thefull readers now reject trailing bytes.
EnvHandle::from_bytes' post-pass).merkle_root_canonical(assumptions)reproducesClaim.assumptionsroots, so claim interop is derivable, not stored.silently changed a Nat/String literal under an otherwise-valid file. Lean
getEnvalso gains the per-const hash check it was missing.
Bundles
Env::bfs_closure— 3-edge traversal:Constant.refs, projection →Mutsblock(projections have empty refs; a refs-only walk returns just the root), and
Mutsblock → member/ctor projection addresses (ingress_anon_blockrequires them).kernel::anon_work::closure_addrsnow delegates to it.Env::prune_to_closure(main, assumed)— builds a closed bundle: value closure cut atassumed(minimal: only cut points actually reached), genuine const bytes, blobs,per-const hints, and display metadata (named entries, name parent chains + string
blobs,
DataValuepayload blobs,meta_refsedges, aux_gen originals).Env::validate_closed()— receiver check: everything reachable frommainiscarried or assumed.
Byte-identical mirrors (bonus fix)
The Lean and Rust writers were never byte-identical: Rust emits the anonymous name as
names-table entry 0, Lean didn't — and nothing ever byte-compared the writers directly.
Lean now matches Rust, which also fixes a latent
putIdxhazard (thegetD 0fallbackused to collide anon references with the first real name). New strict
serEnv == rsSerEnvtests lock this in.Breaking change
All existing
.ixefiles are stale — regenerate withix compile(readers emit a"pre-bundle-format .ixe; recompile it" hint). No golden fixtures existed;
.ixeisgitignored and regenerated per run. FFI
RawEnvctor arity is 5→8 andRawEnvLazy3→5 (the manual builders in
ffi/compile.rsnow populate the new slots — leaving themunset was a segfault).
Verification
--all-featuresclean. New ixon tests: bundleroundtrips through all four readers, blob-tampering rejection ×4, unsorted/dup
assumptions, tampered
main, trailing-garbage vs early-stop, hints-derivation byteparity, projection/Muts closure, prune/validate (offsets in tampering tests are now
computed with the real parsers instead of hardcoded).
lake test(ffi + ixon suites) green; generators produce bundle fieldsand content-addressed blobs.
serEnv== RustrsSerEnv== roundtrip,byte-exact at 505,759,024 bytes (
rust-serializeignored suite).ix compile Tests/MinimalDefs.lean→ new-format.ixe→ix check-rs60,601/60,601 (meta) and 51,681/51,681 (anon);
bench-typecheckexecutes andproves constants over the new format (on top of the Aiur Inductive-flag-drop port).
Follow-ups
prune_to_closurerescansenv.namedper fixpoint round — fine for bundles, worthan index if pruning against mathlib-scale envs becomes common.
Commit.compileDef→ bundle CLI) arefuture work; this PR is the format + helpers.