Local telemetry archives for Maple's embedded store (depends on #129)#194
Open
robbiemu wants to merge 84 commits into
Open
Local telemetry archives for Maple's embedded store (depends on #129)#194robbiemu wants to merge 84 commits into
robbiemu wants to merge 84 commits into
Conversation
Export acquireCheckpointPin / releaseCheckpointPin and withMaintenanceLock from the checkpoint module so the dependent archive branch can pin an immutable checkpoint against retention and serialize its work against checkpoint creation, restore, and reset. Pin acquisition resolves and validates the checkpoint and writes a UUID-named durable pin record under backups/pins/<id>/; release removes only the exact owned record and fails closed on identity mismatch. The maintenance lock wraps the existing sibling ownership lock with the same dead-PID reconciliation and live-PID refusal semantics. Add the archive module skeleton: ArchiveError typed error, the six raw-telemetry signal map with event-time columns, and the archive path model with strict ID and range-date validation, recursive symlink rejection, root containment, and a live-data-directory separation guard. Twelve pin and lock tests cover acquire/release, retention protection, identity mismatch, escape refusal, and concurrent-owner rejection.
Add a validated, documented tuning model for the seven machine-sensitive archive knobs: Parquet writer threads, row-group rows, maximum shard rows, maximum estimated shard bytes, target chunk bytes, minimum free-space reserve, and the archive and scratch roots. Defaults are the measured research baselines (max_threads=1, 10,000-row groups, ~500k rows / ~256 MiB per shard). The parser rejects non-positive or fractional values, a row group larger than a shard, a shard byte budget too small for one row group, a free-space reserve larger than the chunk target, an implausible thread count, and missing roots. A tuning record captures the effective values for every generation manifest so a generation is reproducible and deployment drift is visible.
Add the versioned, strict archive generation manifest and active-pointer formats with location-bound parsing that rejects unknown versions, signal/range/ generation mismatch, negative counts, malformed shard names, and missing tuning. A read helper binds a manifest to its on-disk (signal, range, generation) directory. Add the Parquet shard exporter: one bounded UTC-hour slice per shard, written via SELECT ... INTO OUTFILE FORMAT Parquet directly on the restored scratch chDB (result consumed as a write side effect, never returned into JavaScript). Each shard is validated for row count, byte bound, and SHA-256 before the generation is sealed. Add the generation write lifecycle: acquire maintenance lock, resolve and pin the checkpoint, restore to sacrificial scratch, export bounded shards, validate row counts against the source, durably move the owned building generation into place, atomically select it through the active pointer (reporting supersession while retaining the old generation), append the catalog, release the pin, and remove only owned building output. Free-space preflight runs at operation time. Twenty manifest, pointer, path-model, promotion, supersession, and catalog tests cover the pure and filesystem-level state machine; the full chDB export path is exercised by the native smoke.
Add the archive read-side. listActiveGenerations reports the active generation per signal/range with archived row count, shard count, total shard bytes, and the active generation's Parquet shard paths in order. Superseded generations are retained on disk but never appear in listings or active-path output, so late-arrival history cannot be double-counted. A malformed active pointer for one range is skipped without hiding the others, and the pointer file is left untouched. activeParquetPaths resolves the machine-readable DuckDB-ready shard paths for one signal across all sealed ranges in ascending order, excluding superseded generations. rebuildCatalog regenerates the per-signal catalog.jsonl from the authoritative generation manifests, so a truncated or missing catalog is recoverable without rescanning Parquet bytes. All retained generations (active and superseded) appear once; a generation directory missing its manifest is skipped. Seven listing and catalog-rebuild tests cover active selection, supersession exclusion, cross-range ordering, malformed-pointer tolerance, and catalog rebuild including superseded and manifest-less generations.
Register a 'maple archive' command group with three subcommands following the server.ts Effect-CLI house style: - 'maple archive create <range-date> <signal>': seal one UTC day of one signal into a validated Parquet generation from the current or a specified checkpoint, using the centralized tuning config and reporting supersession. - 'maple archive list [--format summary|paths|json] [--signal <name>]': report active generations with sizes and paths, or emit machine-readable active Parquet shard paths (excluding superseded generations) for DuckDB. - 'maple archive rebuild <signal>': rebuild a signal's catalog from manifests. All commands are local-only by construction (filesystem/checkpoint work, never WarehouseExecutor), map failures to ArchiveError for non-zero exit, and accept the inherited shared flags without acting on them.
Add docs/local-telemetry-archives.md covering the hot store, immutable checkpoint source, pinning, shared scratch lifecycle, Parquet generations, catalog, calibration, and the independent DuckDB query path with memory-limit and spill guidance. Document the happy path from checkpoint through DuckDB, every major off-happy-path outcome (unavailable/incompatible checkpoint, stale pin, interrupted restore, partial shard, validation mismatch, full volume, pointer or catalog corruption, late arrivals, interrupted GC, insufficient budget, failed calibration), which failures leave the live store untouched vs. require action, the capacity/resource model with its one-machine caveat, and the v1 non-goals. Add the archive create/list/rebuild commands to the local-mode CLI reference, cross-linking to the architecture document.
Add native-archive-smoke.sh: ingest markers into all six raw tables, create two checkpoints, archive a sealed UTC day per signal, list active generations, query each archive with DuckDB for exact source counts (2), prove the live store is unchanged after archiving, and rebuild every signal catalog. Two consecutive runs pass. Fix two real defects the smoke exposed: 1. JSONEachRow count parsing. count() results from chDB's FFI path are newline-delimited JSON objects, not a JSON array. The day-count, hour-count, and time-bounds helpers were JSON.parse-ing the whole string as an array and indexing [0], yielding undefined -> 0 rows. Now split on newlines and read the count() column by name, matching the checkpoint module's readJsonRows. 2. Date predicate correctness. chDB's bundled ClickHouse miscounts aggregate count() over a toDateTime64-vs-DateTime predicate (per-row comparison is correct but the aggregate returns zero). Switched all range predicates to toDate()/toHour() equality, which normalizes both second-precision DateTime and nanosecond DateTime64(9) event-time columns. Also rename the archive list format flag from --format to --output to avoid collision with the global --format (json|table) shared flag, and update the docs accordingly.
…nfig Add 'maple archive calibrate' and its internal 'calibrate-run' worker. The calibrator runs a bounded matrix of Parquet writer/row-group/shard candidates against a pinned checkpoint restored into sacrificial scratch, measuring peak RSS, wall time, and output bytes per candidate in a fresh child process (so peak RSS is measured per-candidate, not accumulated in-process). It selects the lowest-RSS candidate within the operator's memory and time budgets, reports high/low confidence, and writes a versioned configuration document only when --write-config is passed (never auto-applies). A native calibration smoke verified the full round-trip: four candidates ran at ~320MB RSS each, the matrix selected writerThreads=1/rowGroupRows=20000 within a 1GB budget with high confidence, and the generated config recorded the measured peak RSS, output bytes, and effective tuning. An impossible budget fails closed with low confidence and no config mutation. Per D-017, true external-volume deployment-scale calibration under the deployment chDB/user/filesystem is a Phase 3 dependency; Phase 2 proves the mechanism and generated-config round-trip on a local scratch volume.
…ocation (C-1, H-7) The prior approval missed a reproducible external-write vulnerability: symlinked archive descendants (signal root, catalog, manifest, pointer) could be followed by mkdir/write, placing state outside the configured archive root. Path safety (C-1): - ensurePrivateDirectory now walks each ancestor with lstat BEFORE creating, refusing to cross a symlink at any depth (mkdir -p + final lstat was unsafe). - A sync assertNoSymlinkSync variant is added for the synchronous read-side. - promoteGeneration, appendCatalog, and rebuildCatalog now assert no-symlink on every path they create or overwrite, immediately before use. - assertSafeRealPath verifies type + containment + no-symlink at read/write sites. Strict state binding and fail-closed malformed handling (H-7): - parseArchiveActivePointer binds signal/range to its directory, rejecting a pointer copied or moved to the wrong range. - listActiveGenerations surfaces malformed pointers/manifests as errors instead of silently skipping them, while still listing unaffected ranges. - Catalog entries now carry formatVersion 1. Eight external-sentinel tests prove the escapes are closed: symlinked signal root, catalog, manifest, pointer, and ancestor; outside targets are verified untouched. Malformed-pointer surfacing and pointer-directory binding are tested.
… review) The R1 fix closed the C-1 write side but left the read side open: a non-racing attacker who plants one symlinked descendant (generation dir, shard file, or range dir) could make listActiveGenerations/activeParquetPaths/rebuildCatalog follow it out of the archive root, feeding attacker-controlled Parquet to DuckDB and attacker-controlled manifest fields to the catalog. The fix's own guard primitives were written but never wired into read paths. This closes the asymmetry: - readArchiveGenerationManifest (the single chokepoint for manifest reads) now asserts no-symlink + real-file before readFileSync, so a symlinked signal/range/generation/manifest chain is refused. - listActiveGenerations asserts no-symlink on the active-pointer path before reading it and on every shard path before returning it to DuckDB. - The dead assertSafeRealPath helper (never called) is removed. Three read-side sentinel tests prove the escapes are closed: a symlinked generation dir is refused by readArchiveGenerationManifest and rebuildCatalog (the attacker manifest is not trusted); a symlinked shard file is refused by listActiveGenerations and surfaced as an error rather than returned to DuckDB. Outside targets are verified unread.
…, fresh-root creation (H-7/R7-R8) Close four defects from the Gate 1 conditional review: 1. activeParquetPaths now THROWS when any relevant range for the requested signal has a malformed pointer/manifest/shard, rather than returning a partial path list that would silently feed DuckDB incomplete data. Errors for other signals do not block a clean signal's paths. 2. rebuildCatalog now PREFLIGHTS every manifest before writing. If any generation is missing its manifest, is malformed, or is on a symlinked path, the existing catalog is PRESERVED untouched and the call throws. The old code silently skipped corrupt generations and wrote a partial catalog. 3. Pointer reads now require a real regular file (assertRealFileSync), matching the manifest-read check, so a non-file entry cannot feed undefined content. 4. ensurePrivateDirectory's root parameter is now MANDATORY (was optional and degenerate when omitted). Fresh archive-root creation is repaired: the root is created first before walking children, fixing the ENOENT on a fresh root. assertRealFileSync is moved to paths.ts as a shared sync primitive. Three new tests cover the fail-closed activeParquetPaths (malformed same-signal range throws; different-signal error does not block), rebuild catalog preservation on missing manifest, and fresh-root nested directory creation.
…dation (H-1, H-2) R3 — bounded sharding: a sealed UTC day is now partitioned by UTC-hour windows, then within each hour by LIMIT/OFFSET sub-splits when the hour exceeds maxShardRows or the estimated uncompressed-byte budget (maxShardBytes). The prior code aborted if a single hour exceeded the bound instead of splitting. Shard names encode their slice: HH-NNNN.parquet (hour + sequence). R2 — pre-promotion Parquet validation: after writing each shard, it is REOPENED via chDB file(path, Parquet) and its row count, min/max event time, and column list are READ BACK from the Parquet file. A 19-byte invalid 'Parquet' file fails this reopen because file() cannot parse it. The shard record now carries the REOPENED counts and columns, not the source counts — closing the tautology where the record always matched the source query. An empty (0-row) reopen also fails. The ArchiveShardRecord gains a columns field (schema round-trip proof); the manifest parser strictly validates it. Test fixtures updated to the new shard name format and columns field.
…ath escaping (CR-1, H-A, H-B, H-C, M-1, M-2) Close the adversarial R2/R3 review's release blockers: CR-1 — sub-sharding now uses ORDER BY (_part, _part_offset) + LIMIT/OFFSET, making each hour's sub-shards an EXACT partition (no overlaps, no gaps). The prior LIMIT/OFFSET without ORDER BY was nondeterministic and could silently duplicate or drop rows. The last shard gets the remainder via min(rowsPerShard, hourRows - offset), and the expected row count is computed and passed to validation. H-A — DESCRIBE file() failures are no longer swallowed. A schema that did not round-trip fails the shard (columns.length === 0 throws), so the manifest's columns field is a real guarantee, not best-effort. H-B — validateShard now checks the reopened row count against the intended slice size AND verifies all rows fall within the expected hour window (toHour(min) === toHour(max) === hour), so a shard containing rows from the wrong hour is rejected. H-C — the byte bound is now enforced on UNCOMPRESSED bytes via parquet_metadata().uncompressed_size, not compressed on-disk size. Compression could previously keep an oversized shard under the compressed ceiling. M-1 — operator paths containing single quotes or backslashes are rejected before export (assertSafePath); both INTO OUTFILE and file() literals use the same sqlLiteral escaping. M-2 — the promoteGeneration test no longer passes a stale 7th argument.
…lash rejection) The adversarial R2/R3 review flagged that no test exercised the path-rejection (M-1) primitives. Add two pure-logic tests proving exportSignalShards rejects a shardsDir containing a single quote or backslash before any chDB query runs. The full Parquet write+reopen+validation path is exercised by the native smoke.
…Gate 2 NO-GO)
Fix two confirmed chDB runtime incompatibilities the native smoke exposed:
- formatRow(RowBinary, *) -> formatRow('RowBinary', *): the bare token RowBinary
is an unknown identifier in chDB (code 47); it must be a quoted string literal.
- parquet_metadata() -> file(path, ParquetMetadata).total_uncompressed_size:
parquet_metadata() is DuckDB syntax, not ClickHouse; bundled chDB exposes
uncompressed size via the file() ParquetMetadata interface.
Strengthen validation to compare against source intent (not just 'parseable'):
- captureSourceSchema captures name+type before export; validateShard compares
the reopened Parquet schema against it (base type match), so a schema that did
not round-trip fails the shard.
- validateShard verifies the UTC DATE of all rows (toDate min==max==rangeDate)
in addition to the hour window.
- estimateBytesPerRow and validateShardBytes now use the corrected chDB APIs.
Strengthen manifest strictness (H-7):
- sha256 must be 64 hex chars; counts must be safe non-negative integers.
- Cross-field: unique shard names, shard-row sum == archivedRowCount,
sourceRowCount == archivedRowCount, rangeEndExclusive == next-midnight.
- parseShardRecord requires nonempty columns; min <= max event time.
Other repairs:
- Strict calendar date validation (reject 2026-02-31; JS Date normalizes it).
- rangeEndExclusive is now the next midnight (exclusive), not 23:59:59 inclusive.
- Free-space preflight checks the actual destination volume (climbs to the
closest existing ancestor when the root is missing) and includes working bytes.
- rebuildCatalog is async and uses durableWrite (atomic temp+fsync+rename) so an
interruption cannot destroy the prior catalog.
- Active shard paths are asserted to be real existing regular files.
- Pin-release failures are surfaced to stderr, not silently swallowed.
133/133 tests pass; typecheck, lint, format clean.
The native smoke exposed three runtime issues after the API fixes: 1. Timezone: toDate()/toHour() without an explicit 'UTC' argument used the host session timezone, causing the export query and the validation query to shard by different hours (12 UTC stored, 08 EDT computed). All toDate/toHour calls now pass 'UTC' explicitly in export.ts and generation.ts. 2. Schema normalization: Parquet export strips LowCardinality(...) wrappers and widens DateTime to DateTime64. The schema comparison now unwraps LowCardinality/Nullable and normalizes DateTime to DateTime64 before comparing base types, so legitimate Parquet type transformations pass while real schema loss still fails. 3. Smoke determinism: the insert markers now use toDateTime64(..., 9, 'UTC') at a fixed UTC noon timestamp within RANGE_DATE, so the archive's toDate/toHour predicates find the rows regardless of the host timezone. The prior now()/ now64(9) inserts landed on a different UTC day when local and UTC diverged. Two consecutive native archive smokes PASS: 6 signals archived, DuckDB exact counts (2 each), live store unchanged, catalog rebuilt. 133/133 unit tests, typecheck, lint, format clean.
…TICAL cursor defect) Replace the broken ORDER BY (_part, _part_offset) LIMIT/OFFSET pagination with a true static-snapshot physical plan that is merge-safe: 1. For each UTC hour, enumerate the active parts and their _part_offset ranges (freezing the layout into a shard plan). 2. Split each part's offset range into bounded intervals (one Parquet file each). 3. Export each shard via WHERE _part = '<part>' AND _part_offset BETWEEN <lo> AND <hi> — a physical predicate targeting exact immutable rows, not a relative OFFSET that drifts when parts merge. 4. After all shards for the hour, re-enumerate and verify every planned part still exists with the same row count. If a merge removed or changed any part, the export fails closed. The prior OFFSET approach was confirmed corrupting data: an injected merge between shard queries produced [5,6,7,8,5,6,7,8], omitting [1,2,3,4] while all local validations and the aggregate count passed. The part-interval plan targets exact rows by immutable location, so a merge of a different part cannot affect this shard's rows, and if THIS part merges away, the post-export verifyPartsUnchanged detects it. A multi-part adversarial native probe (native-archive-merge-probe.sh) reproduces the cross-check's corruption scenario and confirms the fix: 2 parts → 2 shards, exact ID match [t0..t7], no duplicates. The dead estimateBytesPerRow is removed; the byte bound is enforced post-write via parquet_metadata uncompressed size.
…amper detection) listActiveGenerations now hashes each shard file and compares its SHA-256 and byte size against the manifest before returning the path to DuckDB. A tampered regular file whose actual hash or size disagrees with the manifest is rejected (D-015's checksum-mismatch fail-closed rule). The manifest is authoritative; a mismatched file is surfaced as a listing error, not silently returned. Test fixtures updated to compute real SHA-256 and byte size from placeholder content, and to use the new HH-NNNN.parquet shard name format. 133/133 pass.
… comparison Three remaining cross-check blockers closed: 1. Merge freeze: exportSignalShards now runs SYSTEM STOP MERGES before enumerating parts and SYSTEM START MERGES in a finally. Without this, a background merge on the restored scratch store changed part names between enumeration, export, and source-interval validation, causing stale-part false failures (and on the old OFFSET approach, silent corruption). The freeze makes the physical layout stable for the entire export. 2. Source-interval validation: validateShard now re-queries the SOURCE table for the same (_part, _part_offset) interval and compares count + nanosecond time bounds (via toUnixTimestamp64Nano(toDateTime64(...))) with the reopened Parquet. This proves each shard contains exactly the source rows for its physical interval — a wrong-but-valid row set can no longer pass. Time bounds use nanosecond comparison to avoid session-timezone string mismatches. 3. Exact schema comparison: compareSchema now enforces exact column count, name, and order (positional match), plus base-type compatibility (LowCardinality/ Nullable unwrapped, DateTime normalized to DateTime64). Extra/dropped/ reordered columns fail closed. Nested type parameters (Map key/value types) are still normalized to the base token — documented as a compatibility limitation. Also: ExportSettings gains an afterShardValidated callback for the adversarial merge-injection probe. The sha256FileSync comment is corrected (reads whole file, not streaming). verifyPartsUnchanged is replaced by verifyHourTotalUnchanged (robust to part-name churn from merges, detects data loss/gain). Two native smokes PASS: six-signal (with source-interval validation on all six tables) and merge-safety (2 parts → 2 shards, exact ID match). 133/133 unit tests.
…apleTechLabs#5) A standalone probe that reproduces the exact cross-check corruption scenario: two parts for the same UTC hour, with an OPTIMIZE TABLE ... FINAL injected between shard exports via the afterShardValidated hook. SYSTEM STOP MERGES blocks the OPTIMIZE (Code 236, Cancelled merging parts), the physical layout is frozen, and the exported shards contain the exact source row set [t0..t7] with no duplicates or omissions. This is the definitive proof that the static-snapshot part-interval plan is merge-safe: the prior OFFSET approach produced [5,6,7,8,5,6,7,8] under the same scenario. The probe uses the real bundled chDB via FFI (MAPLE_LIBCHDB).
added 5 commits
July 9, 2026 08:14
…mparison
Closes finding 3 (held-out contract) on top of the round-2 repair. The held-out
sample is now strictly LARGER than training and disjoint: training covers ordered
rows [0, sampleRows); held-out covers [sampleRows, sampleRows + 2*sampleRows)
(heldOutSampleRows = HELD_OUT_SAMPLE_MULTIPLIER * sampleRows). Every training and
held-out result carries a persisted sample scope
({checkpointId, checkpointManifestFingerprint, rangeDate, role, startRow,
requestedRows, rowCount}) emitted by the child and attached by the parent; the
config records a samplePolicy and the loader proves every scope binds to ONE
checkpoint/range and that the two windows are disjoint and correctly sized.
Hybrid predicted-vs-observed comparison (per the calibrator contract): because a
2x-larger held-out naturally takes ~2x wall time and bytes, wallMs and
physicalBytes predictions are rescaled by heldOut.logicalBytes /
training.logicalBytes before the two-sided check; throughput and compressionRatio
(size-invariant rates) compare directly; peak RSS and peak temp disk compare as
absolute peaks (NOT scaled by row count — they are process peaks, not per-row
costs). The raw values, scale ratio, training/held-out logical bytes, adjusted
predictions, and comparison results are all persisted; the loader recomputes the
ratio and every comparison from the recorded evidence using the fixed canonical
tolerances only (no run-specific widening). comparePredictedObserved gains an
optional sizeScaling param mirrored by the loader's expectedComparisons.
Native calibrate probe asserts the larger persisted held-out scope and
samplePolicy for every result (training=10 held-out=20, disjoint, single source)
and the loop closes through the hybrid comparison. Unit tests add hostile
forged-scale-ratio and forged-scope cases plus a held-out-larger-and-disjoint
property. 306/306 unit; typecheck/lint/format/diff-check clean.
Closes the final D-026 contract: the held-out comparison is now PER-SIGNAL and
like-for-like, replacing the cross-signal aggregate ratio that could mask a
single-signal regression.
Each signal's held-out result is paired with the same candidate's TRAINING
result for that signal (compareHeldOutPerSignal). wallMs/physicalBytes are
scaled by THAT signal's own heldOut.logicalBytes/training.logicalBytes ratio;
throughput and compressionRatio are compared directly (size-invariant rates);
peak RSS and peak temp disk are compared as absolute peaks. The attempt passes
only when ALL SIX signals pass; cross-signal aggregate extrema (worstCase) are
recorded only as a descriptive summary and never decide acceptance.
Data model (config format stays v2): heldOut and each heldOutAttempts entry
carry signalComparisons [{signal, scaleRatio, comparisons, passed}] in canonical
six-signal order — no duplicated raw metrics (the loader re-derives pairs by
exact candidate + signal identity). A complete attempt has six entries even
when it fails; an incomplete/over-budget/short-window attempt has
signalComparisons: [], worstCase: null, passed: false. Training and held-out
logicalBytes must both be strictly positive (an undefined ratio makes the
attempt incomplete, never silently ratio 1).
The loader recomputes every per-signal ratio, adjusted prediction, relative
delta, and pass flag via expectedSignalComparisons and requires exactJson
agreement. The runner-level compareHeldOutPerSignal is pure and unit-tested
with the reviewer's exact cross-signal counterexample (signal A regresses 1.0
under per-signal but aggregate reports delta 0). Hostile cases updated for the
per-signal shape (forged comparison, forged scaleRatio, forged canonical
tolerances with recomputed per-signal comparisons).
calibration-validation-compare.ts (probe reference) now imports canonical
HELD_OUT_TOLERANCES and applies the per-signal logical-byte ratio. Native probe
asserts six signalComparisons in canonical order, each with its own ratio.
R9 hybrid section documents signalComparisons, per-signal scaling, and the
complete-vs-incomplete shape. 309/309 unit; typecheck/lint/format/diff-check clean.
Contributor
|
Your Pullfrog Router balance is exhausted. You have a card on file but auto-reload is disabled, so runs paused once your balance went past the overdraft buffer. Top up balance → · Enable auto-reload →
|
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
Makisuo
requested changes
Jul 11, 2026
Collaborator
There was a problem hiding this comment.
Thanks for putting this together. It goes way further than I would have gotten myself with Maple Local, but from looking more into this, this for sure makes sense to add, looks extremely usefull. The state-machine and filesystem coverage here is strong, but I don't think this is safe to merge yet. The main blockers are the fresh-process checkpoint reopen failure and several calibration paths that currently fail open on invalid inputs or child output.
Please fix the inline findings, make the full create -> restore -> fresh-process reopen lifecycle repeatably green, and wire the native checkpoint/archive/recovery probes into required CI. This also needs to land after #129 and be brought up to date with main before merge.
Contributor
Author
glad to contribute, thank you for this amazing local Otel! is 129 on track to merge? edit: ooh, I see, you've added a problem for that PR here. looking to reproduce |
added 11 commits
July 11, 2026 23:57
# Conflicts: # apps/cli/src/bin.ts
Contributor
Author
|
Ready for re-review. All requested review threads are addressed and resolved, the checkpoint fix landed on the dependency branch and is merged here, the branch is current with main, and checkpoint/archive/crash-recovery native probes are now wired into PR CI. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Local telemetry archives for Maple's embedded store
Summary
This PR adds long-term, portable Parquet telemetry archives to Maple's local
mode. It exports the six raw telemetry tables (
logs,traces,metrics_sum,metrics_gauge,metrics_histogram, andmetrics_exponential_histogram) fromvalidated checkpoints into immutable UTC-day generations that can be queried
directly with DuckDB.
Archives do not reload history into the live store, add a live export endpoint,
or require a second always-on database.
What changed
archive createexports one signal and UTC day from restored checkpointscratch storage and seals a validated Parquet generation.
archive listexposes active generations or verified shard paths for DuckDB;archive rebuildreconstructs catalogs from manifests.pointer. Superseded generations remain retained until safe GC.
archive gcuses ownership-aware tombstone retirement and crash recovery.archive calibratemeasures the deployment's real export behavior and canwrite a hash-bound tuning configuration.
opens or mutates the live chDB store.
Safety model
Export restores one validated checkpoint into private sacrificial scratch
storage. A persistent checkpoint pin is acquired under the maintenance lock,
protecting the source from retention while export runs. Checkpoint operations,
export, calibration, reset, restore, and GC serialize through that lock.
Generations are immutable. A late-arrival re-export supersedes an older
generation structurally rather than attempting unsafe cross-table
TraceIddeduplication. Manifests bind source checkpoint identity, shard hashes, row
counts, event-time bounds, schema identity, and effective tuning.
Calibration compatibility
New calibration documents are
formatVersion: 3and use directional resourcecost evidence: lower observed RSS, wall time, compression ratio, physical bytes,
or temporary-disk peak is safe; only a regression beyond tolerance fails.
The loader remains compatible with immutable
formatVersion: 2operatorartifacts. A v2 document is accepted only if its complete held-out evidence
matches one coherent historical policy:
repair.
Format 3 accepts directional evidence only. A document mixing policies across
selected evidence or held-out attempts is rejected. Manifest generation format
3 preserves the loaded tuning identity, including v2 and v3 config identities.
Calibration child processes write
/usr/bin/timeoutput to a private per-childreport and wait for
closeso worker stdout/stderr is fully drained before thenext candidate starts.
Operational workflow
archive calibrate <day> --write-config <path>.archive create <day> <signal> --config <path>for each signal.archive verifyto stream and verify active shard contents.archive list --output pathsto obtain the verified active Parquet paths.read_parquet(..., union_by_name=true).The source row count, archived row count, and sum of shard row counts must
agree before a generation is published.
Validation evidence
Source validation for the current hardening commit:
git diff --check: pass.5d06ba124c36304754aed2c0960a8a17aad6db762b078ce6b13a15f3a8b1c88b) loadsunchanged with format-2 identity and calibrated overrides. Its 17 lower-cost
resource comparisons all use directional
relativeDelta: 0evidence.Previously deployed Phase 3B validation on the r4 predecessor passed the
archive lifecycle, calibration, config-bound creation, and DuckDB verification.
The DuckDB check read generation
17d4f48aacross two shards and found bothmarker rows. A later config-bound archive sealed generation
f3166830-299f-4396-8c04-619e9b201ffcwith 35,112 log rows across four shardsand a hash-bound tuning identity.
Deployment status
The deployed r4 image remains the proven Phase 3B artifact. Commit
269053b4is source-only compatibility and timing hardening; it has not yet been built
or deployed. A fresh image build and native probe are required before claiming
the current commit itself is deployment-proven.
Known limits and follow-up
checkpoint-pause characterization.
UI.
Updates
Devin review follow-up — 2026-07-10
ArchiveErrorchannel rather than throwing insideEffect.sync.git diff --checkpass.1378ecce.Review follow-up and checkpoint restore stabilization 2026-07-12
The archive review exposed an intermittent failure while reopening checkpoints through chDB’s scratch
RESTORE DATABASEpath. We reproduced it independently in the checkpoint implementation, addressed it there first, and then merged the fix into this branch.Investigation showed that chDB v26.1.0 uses a separate 16-thread restore pool even when table loading is serialized. Timing delays and database-readiness probes did not reliably eliminate the failure, while serializing restore workers with
--restore_threads=1survived the expanded checkpoint and archive stress tests.This deliberately trades restore parallelism for reliability in Maple’s embedded workload. The 16-thread value is chDB/ClickHouse’s normal default, and the precise upstream mutex defect remains unidentified; the evidence does not establish that multithreaded restore is universally unsafe.
Combined validation note
At the current-main-merged #194 head, validation completed successfully:
git diff --checkAll seven GitHub review threads have replies and are resolved. The review fixes, updated checkpoint dependency, current
main, and required native archive/recovery CI are committed locally and ready for the planned combined push.