PRD: SACC data format + Smokescreen blinding

[cailmdaley]

PRD: SACC data format + Smokescreen blinding

Design document for issue #234 (SACC + blinding design), under epic #232 (SACC data format + Smokescreen blinding). The work is organized as a series of PR-sized rows, tabulated in §8; each sp_validation row already carries an open draft PR — discussion first, code second — while the two fork PRs are not yet open.

Summary

For the tomographic round — the UNIONS tomographic cosmic-shear analysis, the successor to the 2D shear paper — sp_validation adopts SACC (LSST DESC's data-vector container) as the standard format for all data products, and blinds the data vector with a UNIONS fork of Smokescreen (DESC's implementation of Muir et al. 2020 data-vector blinding), adapted so that theory comes from a pluggable backend with CCL as the shipped default.

Decisions — treated as settled unless someone objects:

• One SACC file per catalogue version holds everything: ξ± on the coarse grid, pseudo-Cℓ with bandpower windows, COSEBIs, n(z) tracers, covariance, ρ/τ PSF diagnostics as custom data types, and the fine-grid ξ± COSEBIs-integration input as grid='fine' tagged points with its own dense covariance block (§1).
• Smokescreen is forked to UNIONS-WL/Smokescreen and given a theory_fn protocol: any callable that maps cosmological parameters to a theory vector can drive blinding, with a built-in CCL backend as the default. The firecrown integration is inherited from upstream and unsupported in the fork, and sp_validation never installs it (§3).
• The blind shifts the measured ξ± and pseudo-Cℓ by a theory difference — hidden cosmology minus fiducial — with the hidden cosmology drawn uniformly within ±0.075 in S8 and ±0.1 in Ωm (§4).
• Blinding is per intermediate product, at birth: the seed is drawn once per catalogue version, each blindable part (coarse ξ±, fine ξ±, pseudo-Cℓ) is blinded as soon as it is computed, and only blinded intermediates persist on disk — downstream work never waits on a terminal blind. COSEBIs and pure-E/B are derived from the blinded fine ξ±, never shifted directly; the covariance is never touched — Smokescreen encrypts the true vectors and we delete the plaintexts (§4).
• Custody is a hash commitment: a script draws the seed, blinds immediately, publishes hash(seed); unblinding reveals the seed and anyone can verify it against the commitment (§5).
• CosmoSIS keeps running unchanged behind a SACC→2pt-FITS converter; adopting CosmoSIS's native SACC likelihood is its own row (§8), validated against the converter (§2).
• Issue Format of the text file obtained from the redshift distribution. #228 (which n(z) normalization convention to adopt) closes: every consumer normalizes n(z) internally, so no convention is needed (§1).

Open questions for the group:

• The shift envelope: is ±0.075 in S8 the right size? (§4)
• The fiducial cosmology and model configuration (halofit version, IA model, neutrino treatment) — one named choice that both theory stacks implement. Implementation proceeds on reasonable defaults meanwhile; the choice changes config values, not code (§6).

Deferred: unblinding criteria (drawn from #233 — the enumeration of validation tests for the tomographic round — once it stabilizes) and procedural conventions like hidden contour axes — team process, not code (§7).

1. SACC as the standard format

Today we scatter data products across TreeCorr .txt, custom FITS, .npz, and ASCII files, and hand-assemble them into a "2pt FITS" file — the format DES built for its twopoint library, which CosmoSIS's 2pt_like module reads.
SACC replaces all of it: self-describing files that carry every statistic, its binning, its tracers, and its covariance, in a format the wider community (DESC, HSC) reads natively.

Per catalogue version, one file — {version}.sacc: ξ± on the coarse analysis grid, pseudo-Cℓ with their bandpower windows, COSEBIs E/B modes (SACC defines galaxy_shear_cosebi_ee/bb officially), n(z) as one tracer per tomographic bin named source_0, source_1, … (the suffix CosmoSIS's SACC reader expects), the covariance, the ρ/τ PSF diagnostics and pure-E/B modes as custom data types, and ξ± on the fine θ grid (the binning COSEBIs integration needs) as grid='fine' tagged points — SACC accepts custom types, and one unified file beats a sidecar. In principle coarse is a weighted rebin of fine, but the exact rebin needs per-bin pair weights, so both are measured and stored. The PSF diagnostics stay out of the blind (§4).

The covariance is block-diagonal across statistics (the same independence assumption the current pipeline's assembly makes): the analysis blocks first, then a dense per-pair fine-ξ block — the Gaussian analytic covariance at the integration binning (OneCovariance; Paper II used CosmoCov), which downstream error propagation consumes (pure-E/B errors are MC-sampled from it, the COSEBIs covariance is a matrix transform of it); TreeCorr varxip/varxim diagonal as the degraded fallback. At the production fine binning (1000 θ bins, 0.5–300′) the dense block is ~32 MB per pair; extreme convergence-check grids (the one-off 10 000-bin run) degrade to the diagonal, not to a second file.

Two conventions, fixed here:

• θ in arcminutes. SACC stores θ unitless; we already write arcminutes everywhere.
• n(z) needs no normalization convention. CosmoSIS, OneCovariance, and CCL each normalize n(z) internally (verified in source) — closing Format of the text file obtained from the redshift distribution. #228.

The writers live in a new sp_validation.sacc_io module, and the cosmo_val mixins call it in place of their per-statistic writers.
New products — including new mocks — are born in this format; we regenerate nothing old.

2. Interfaces to other codes

CosmoSIS. A SACC→2pt-FITS converter reproduces today's assembled file exactly, so the whole inference chain — including the PSF ρ/τ likelihood modules in the cosmosis-standard-library fork of Sacha Guerrini (CosmoStat, UNIONS inference) — runs untouched, and we can byte-compare the converter's output against a current file to validate it.
CosmoSIS also ships a native SACC likelihood (likelihood/sacc/sacc_like.py, feature-equivalent to 2pt_like: scale cuts, data-set selection, Hartlap/Sellentin); its Cℓ path carried the HSC Y3 analysis, but its ξ± path has not carried a published analysis and is far less exercised.
So: converter first, native adoption as its own row, each path validating the other.

OneCovariance. It computes covariances rather than consuming data vectors, and it knows nothing of SACC.
The interface is two small pieces of glue: our n(z) tracers → its per-bin two-column text input plus config, and its output covariance list → the SACC covariance block.

3. The Smokescreen fork

Blinding runs through UNIONS-WL/Smokescreen, a fork of DESC's Smokescreen adapted to be theory-backend independent. What the fork exposes:

• A theory-function protocol. theory_fn(cosmo_params) -> np.ndarray, aligned element-by-element to the SACC rows being concealed, is the single theory entry point. Any callable satisfies it; alignment to the data vector is the backend's contract.
• A built-in default CCL backend, so a standard cosmic-shear SACC file blinds out of the box; power users override with their own callable. import smokescreen imports no theory backend — pyccl loads only when the default backend is constructed.
• A fixed pure draw. The hidden cosmology is drawn by a local, order-independent RNG seeded from the blinding seed, over CCL-native parameter names (sigma8, Omega_c, …); the draw is exactly reproducible from the seed and never touches global RNG state.
• Declared packaging. The fork's pyproject declares pyccl alongside the inherited cryptography and sacc>=0.12; Python ≥3.12; no numpy upper bound. Install identity is a pinned git+https tag; the fork keeps the Smokescreen name and is never published to PyPI.
• Firecrown status. The firecrown integration is inherited from upstream and unsupported in the fork: lazily imported, never a declared dependency, not installed, tested, or maintained. sp_validation never installs firecrown; its only theory path is CCL.

The fork work is two PRs on UNIONS-WL/Smokescreen: the protocol + default CCL backend + draw refactor + test-suite migration (#2), and packaging + install identity + CI + pip-only docs (#3). sp_validation's dependency PR pins a fork tag that is cut only after both fork PRs have merged — the lazy firecrown import lands in the protocol PR, and a tag without it would still pull firecrown transitively on a clean install — so the dependencies row (§8) is blocked on the tag, and the tag on both fork PRs. The fork diff is kept small and clean to preserve the option of contributing changes back upstream.

4. Blinding

How it works.
You hand ConcealDataVector the true data vector (a SACC file), a fiducial parameter mapping, a shift range per cosmological parameter, a seed, and (optionally) a theory_fn.
It seeds the fork's local RNG (§3) to draw a hidden cosmology within the shift range, computes theory at both cosmologies through the theory_fn, and writes a new SACC file with the difference added on: d → d + t(hidden) − t(fiducial).
Noise and covariance stay untouched.
The measured vector then points at a cosmology offset by an unknown amount, and nobody can read off S8.
Goodness of fit survives to first order: if the fiducial equals the truth, the shifted data is exactly a realization of the hidden cosmology; otherwise a second-order χ² residual appears, proportional to (truth − fiducial) × (shift size), which Muir et al. show stays negligible for realistic shifts — one more reason not to make the shift range arbitrarily wide.
Unblinding means revealing the seed — anyone can recompute the shift and subtract it — and because the shift is exactly reproducible from the seed, re-blinding an updated catalogue takes seconds and needs no human in the loop.

Blinding is applied per part, at birth. The seed is drawn once per catalogue version (a blind-init step publishes hash(seed) and encrypts the seed); each blindable intermediate — coarse ξ±, fine ξ±, pseudo-Cℓ — is then blinded the moment it is computed, so only blinded intermediates persist on disk and downstream work proceeds as parts land instead of waiting for every product before a terminal blind. COSEBIs and pure-E/B are derived from the already-blinded fine ξ± — born blinded, never shifted directly. The deterministic draw makes all parts mutually consistent (same seed → same hidden cosmology), every blinded part carries the public commitment hash in its metadata, and the terminal assembly asserts the hash is identical across parts before combining.

Three sp_validation theory backends.
The fork's default CCL backend does not know the layout of our analysis SACC file (§1) or our IA configuration, so the blinding step supplies its own theory_fn callables — plain CCL functions that compute exactly our fiducial model — one each for the coarse ξ±, the fine-grid ξ±, and the pseudo-Cℓ (bandpower window applied to ΔCℓ_EE). All three run through one ConcealDataVector draw, so a single hidden cosmology shifts every statistic consistently; a cross-backend test checks that the same hidden cosmology through two callables agrees to machine precision.

Extract → conceal → merge.
The to-be-blinded block (ξ±, pseudo-Cℓ) is extracted from the analysis file, concealed, and merged back; derived statistics — COSEBIs, pure-E/B — are then re-derived from the blinded ξ± through the pipeline's own estimators, never shifted directly. Catalogue-level work — PSF diagnostics, ρ/τ, null tests, calibration, most of #233 — sits upstream of the data vector and never sees the blind.

We only analyze blinded files.
Smokescreen encrypts the true vector at blinding time; we delete the plaintext.
No convention has to remind anyone not to plot the true vector, because the true vector isn't there.

The shift envelope.
Uniform draws in (Ωm, S8) with half-widths ±0.075 in S8 and ±0.1 in Ωm. The fork's draw speaks CCL-native parameters (σ8, Ωc), so the blinding step translates this (S8, Ωm) envelope into the equivalent ranges on those parameters.
The envelope must be at least as wide as the effect we most need to stay agnostic about — the ~0.05–0.075 S8 tension between weak-lensing surveys and Planck — or a tension-sized pull could be reasoned away as "too big to be the blind."
S8 carries the amplitude blinding; the Ωm width just keeps the implied σ8 physical.
Whether to resize is an open question for the group.

Acceptance.
The shift is a theory difference and therefore pure E-mode, so B-mode null tests must give the same answer blinded or not; the acceptance test (agreed with Sacha Guerrini) verifies exactly this on mocks — B-mode estimators unchanged under blinding to the estimator's numerical floor. A second acceptance line: the fork blinds a real sacc_io file end-to-end.

5. Secret custody

The shift is a deterministic function of (seed, config) — feed the same seed back through Smokescreen and the identical shift comes out — so re-blinding an updated catalogue needs no human in the loop, and unblinding is mechanical.

Custody is a hash commitment, with no keyholder:

1. A script draws the seed from OS entropy and applies the blind immediately; no one ever sees the seed.
2. It publishes sha256(seed) plus a digest of the blinding config to the repo — the commitment binds both.
3. It encrypts seed and true vector (via smokescreen.encryption) and deletes the plaintext; the seed is stripped from the blinded file's metadata, which is stamped concealed=True plus a label naming the blinding campaign, so blinded files from different blinds are distinguishable.
4. At the unblinding event the script reveals the seed; anyone recomputes the shift and checks the hash matches what we published — proof the revealed seed is the one fixed at blinding time, not a substitute. The unblind step verifies the hash before subtracting.

Both the blind and the unblind are publicly verifiable, with no trusted party.

6. Theory consistency

The blinding shift is computed through CCL while inference runs CosmoSIS + CAMB, so the shift only means what we intend if the two stacks agree on our binning. Two work items:

• Pick the fiducial cosmology and model configuration — halofit version, IA model, neutrino treatment — as one named choice for the tomographic round, which the CCL blinding backends and the CosmoSIS pipeline both then implement. Needed with or without blinding. We implement and test on reasonable defaults now; the eventual group choice changes config values, not code.
• A CAMB↔CCL cross-check test, part of the blinding row's test suite: same cosmology, same n(z), compare ξ± on our θ grid against a stated tolerance. Two subtleties it settles: CCL takes σ8 as its amplitude input while the CAMB pipeline is parameterized in A_s, and the test must match on σ8 (a nominal A_s leaves σ8 ~3% off and blows the comparison to ~10%); and the halofit_version strings must match on both stacks (mead2020 vs mead2020_feedback differ by several % at k≳1).

7. Deferred

• Unblinding criteria — which tests must pass, at what thresholds, before we unblind — wait for the validation-test enumeration in Enumerate validation tests and null checks #233 to stabilize, then get drawn from it.
• Procedural conventions (hidden contour axes, no-external-comparison rules, post-unblinding change logs) are a team-process discussion that touches no code, and it's not obvious we need them at all once the data-vector blind hides the answer.

8. Implementation series

Two repos: UNIONS-WL/Smokescreen (the fork, umbrella issue UNIONS-WL/Smokescreen#1) and CosmoStat/sp_validation (everything else). The implementation tasks are this issue's sub-issues (listed below with their progress) — one per PRD row, each with its draft PR attached via the Development link; the sub-issue is the single source of truth for its row's scope and status.

Ordering: the dependencies row is blocked on both fork rows — sp_validation pins a fork tag that is cut only from the merged tip of the protocol and packaging PRs (the lazy firecrown import lands in the protocol PR; a pin predating it would pull firecrown transitively on a clean install). The pure-format rows (sacc_io, converters, migration) are independent of the fork and proceed in parallel; the blinding row depends on both fork rows plus the writers.

— Fable on behalf of Cail.