Read the 2-BM file contract, and ask staff only what reading cannot answer - #584
Merged
Conversation
Stage 0 for data ingest. Before writing an adapter that reads 2-BM scan files, establish what those files actually are. The storage chain was already settled by DATA-1 through DATA-7 (#270), but those answers described the journey a file takes, never the file. Reading tomoscan's 2-BM subclass and dmagic answers most of it, and one answer would have cost real work to discover later. A finished capture is NOT a finished file: the end-of-scan sequence stops the file plugin and waits for Capture_RBV to reach 0, and only then calls add_theta(), which reopens the HDF5 in append mode to create /exchange/theta. The obvious ingest trigger, checksum on capture-complete, would checksum a file that is about to change. The file is final after the transfer step reports through ScanStatus. The rest, now recorded rather than assumed: the layout is Data Exchange, named verbatim in the code it post-processes with; files follow the areaDetector template %s%s_%3.3d.h5, which is where the _rec/ basename comes from; dropped frames are detected by add_theta() and only warned about, so a reader that records what landed will silently under-describe a short scan; and the experiment folder is COMPUTED by dmagic as {year_month}-{pi_last_name}-{gup_number} from scheduling data, with a clean_entry() normalisation CORA must reproduce exactly or it will miss on accented surnames. That last one matters most for design: the folder is derivable, not guessable, which is what lets a read-only CORA find a scan without the beamline changing anything for it. Cuts the staff questions from four to two. Three were answerable by reading, and spending an operator's attention on a question you could look up is the wrong trade. What remains cannot be read from source: how often dropped frames really happen, and whether the deployed tomoscan matches the public repository. Also removes fixtures that described storage the beamline does not have, found while checking what CORA already believed. A fictional s3 bucket in four places, and worse because it looked right, a path fixture that had the host and the first three segments correct but a date-only experiment folder, dropping exactly the PI and proposal segments that make the folder derivable. Anyone building path derivation against it would have built the wrong thing with no reason to doubt it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Five deferred infrastructure rows (container orchestration, secrets management, TLS, backup repository target, background scheduler) all gate on one unanswered question, phrased differently in each. That question had never been put to anyone: questions.md carries the hexapod chain, the BLEPS list and the storage pipeline, and nothing about the machine. Deliberately not asking "will it be a virtual machine". That is a preference, and staff would reasonably bounce it back asking what we want. It is also the wrong axis: VM versus bare metal barely reaches CORA, which ships a linux/amd64 image that runs the same on all of them. What the rows actually gate on is single host versus orchestrated cluster, and even a firm answer there leaves the expensive parts open. So every row reduces to a fact only the beamline holds. Two are worth their place on their own: HOST-1 because EPICS Channel Access discovers PVs by UDP broadcast on the local subnet, so a host one routed hop from the IOCs finds nothing and reports it as a timeout rather than an error. It is one configuration line once someone names the gateway, and a day of debugging that looks like a CORA fault until they do. HOST-3 because it carries a deadline rather than a trigger. A backup written to the same disk as the database protects against operator error and corruption and nothing else, and pgBackRest fixes repository encryption when the repository is created, so deferring the target long enough decides encryption by accident. Priorities stay Blocks-go-live: none of these stop CORA describing 2-BM, all of them stop it running there, which keeps the page's standing claim that no build blockers are open accurate. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Coverage report
This PR does not seem to contain any modification to coverable code. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two commits of 2-BM deployment documentation. They ship together because both touch
questions.mdand would conflict as separate branches.Read the file contract instead of guessing at it
Data ingest needed to know what a finished scan actually looks like on disk. Rather than draft questions for beamline staff, this reads the upstream sources: tomoscan's 2-BM subclass and dmagic. Recorded in
operations.md:Capture_RBVto reach 0 and only then callsadd_theta(), which reopens the file in append mode to create/exchange/theta. A checksum taken at capture-complete describes a file that is about to change. The file is final after the transfer step reports throughScanStatus.add_theta()compares written frames against commanded angles and only warns. A reader that records what landed will silently under-describe such a scan.{year_month}-{pi_last_name}-{gup_number}with a specific NFKD-to-ASCII normalisation. Derivable rather than guessable, which matters under a read-only posture.Three of four drafted staff questions turned out to be answerable this way and were dropped. Only DATA-8 (dropped-frame frequency) and DATA-9 (whether the deployed tomoscan matches public HEAD) survive, because neither is in the source.
Ask 2-BM where CORA runs
Five deferred infrastructure rows gate on one question that had never been asked.
questions.mdcarries the hexapod chain, the BLEPS list and the storage pipeline, and nothing about the machine.Deliberately not asking "will it be a virtual machine". That is a preference, not a fact, and it is the wrong axis anyway: VM versus bare metal barely reaches CORA, which ships a
linux/amd64image that runs the same on all of them. Every row instead reduces to something only the beamline can state.Two are worth their place on their own. HOST-1 because EPICS Channel Access discovers PVs by UDP broadcast on the local subnet, so a host one routed hop away finds nothing and reports it as a timeout rather than an error: one configuration line once someone names the gateway, and a day of debugging that looks like a CORA fault until they do. HOST-3 because it carries a deadline rather than a trigger, since pgBackRest fixes repository encryption at stanza creation.
Priorities stay
Blocks-go-live. None of these stop CORA describing 2-BM; all of them stop it running there. That keeps the page's standing claim that no build blockers are open accurate.🤖 Generated with Claude Code