Turn a finished scan file into the record, atomically - #585
Merged
Conversation
The Data BC's machinery was complete and format-free; what was missing was the one adapter that reads a real file. This lands the scan-ingest seam designed and gate-reviewed in project_scan_ingest_design: four independently varying axes (layout, finality, correlation, transport), one port, one adapter, one composing slice. ScanReader speaks frame roles, counts, angles, and structural completeness; /exchange/* and image_key die at the adapter. The role vocabulary carries invalid because NXtomo marks it (image_key=3), and the Data Exchange adapter reports zero meaning none-marked. Shortfall stays distinct from invalid, and dropped_frame_count is optional: absence means the source cannot know (a crashed file has no commanded counts, NXtomo never does), zero means verified-none. Commanded counts come from /process/acquisition, written on clean close by the file's second author, the areaDetector layout XML; the /defaults frame ids alone can never show tail truncation because a missed trigger never receives an id. Flat and dark expectations use tomoscan's own arithmetic: configured count times the phases the mode names. The adapter opens read-only with HDF5 locking disabled, for two reasons: flock is unreliable on the network filesystems the analysis tier lives on, and a reader holding a shared lock can make tomoscan's own add_theta append reopen fail, a failure tomoscan swallows, so observation would silently break the pipeline it observes. ingest_scan composes the register_dataset, register_distribution, and record_acquisition DECIDERS and persists all three genesis streams in one append_streams transaction. It never chains those slices' handlers: each handler is its own transaction, and a mid-chain rejection would strand a Dataset without its Acquisition in a system whose pitch is trustworthy provenance. Any refusal or rejection leaves zero events. The composition core lives at the BC root (cora.data._ingest) because slices must not import siblings; the conductor set that precedent. Re-ingest refuses on a checksum natural key, naming the existing dataset id: register_dataset mints a fresh UUID per call, so nothing else stops double-ingest from silently duplicating the chain. The checksum rather than the uri because the same bytes are legitimately multi-homed across tiers. No queryable checksum surface existed, so the dataset summary projection gains the column pair, an index, and a migration, with the boot gate's version pin moving in the same commit. captured_at never comes from the ingest clock. A parseable file start_date wins; with none, the operator supplies it as the same caller-asserted provenance the base slice already accepts, or the ingest refuses naming the remedy; supplying both is refused as ambiguous. CORA records claims, it does not manufacture them. Ships with an erratum the design work surfaced: operations.md claimed the ScanStatus transfer message marks when the analysis-tier copy exists, but both transfer paths are fire-and-forget at tomoscan HEAD, so the signal marks initiation and arrival must be verified independently. DATA-9 widens to the deployed layout XML and DATA-10 asks the timezone of the timestamp PV. Scan enters the glossary and the slice-subject allowlist as the one deliberate convention extension: the slice's subject is the external artifact itself, before any record of it exists. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Coverage report
Click to see where and how coverage changed
This report was generated by python-coverage-comment-action |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stage 1 of the scan-ingest seam, implementing the gate-reviewed design (four axes: layout, finality, correlation, transport; one port, one adapter, one composing slice). The Data BC's registration machinery was complete and format-free; this lands the one adapter that reads a real file, and the slice that drives it.
What it does
POST /scans/ingest(and theingest_scanMCP tool) takes a locator plus the facts only the caller knows (producing Asset, storage Supply, optional Run), and:ScanReaderport;/exchange/*andimage_keyvocabulary dies at theDataExchangeScanReaderadapterChecksumComputersurface (the posix adapter grew a second protocol over one shared root-confinement helper)append_streamstransaction; a mid-composition rejection leaves zero eventsDesign decisions with teeth
Finality is respected, not assumed.
/exchange/thetapresent means post-processing completed; the ScanStatus transfer message marks transfer initiation (both tomoscan paths are fire-and-forget), so this PR also carries an erratum to the 2-BM operations page that previously claimed the signal marks arrival.captured_atis never fabricated. A parseable filestart_datewins; with none, the operator supplies it as the same caller-asserted provenance the base slice already accepts, or the ingest refuses naming the remedy; both at once is refused as ambiguous.captured_at_sourceis recorded in evidence either way.Counts distinguish zero from unknowable.
dropped_frame_countis optional: absence means the source cannot know (a crashed file has no commanded counts; NXtomo never does), zero means verified-none. Commanded counts come from/process/acquisition, written on clean close by the areaDetector layout XML, because the/defaultsframe ids can never show tail truncation.The reader cannot break the beamline. HDF5 opens read-only with
locking=False: a reader holding a shared lock can make tomoscan's ownadd_thetareopen fail, a failure tomoscan swallows, so observation would otherwise silently break the pipeline it observes.Supporting changes
20260729120000adds checksum columns + index to the dataset summary projection (no queryable checksum surface existed), with the schema boot gate's version pin moved in the same commitScanenters the glossary and the slice-subject allowlist as one deliberate convention extension, rationale inconventions.mdcora.data._ingest) per the conductor precedent, since slices must not import siblingsdata:section (verified facts only:layout,finality) plus its schema; DATA-9 widens to the deployed layout XML; DATA-10 asks the timestamp PV's timezoneVerification
17 reader tests on synthetic Data Exchange files, 13 handler tests pinning zero-events-on-every-refusal, 5 contract tests, integration tests landing a real HDF5 file as three streams in real Postgres and refusing a duplicate via the new columns, all 952 deployment-descriptor tests, full tiers green (45,757 fast + 1,084 integration/e2e), docs build clean.
🤖 Generated with Claude Code