Image-simulation m-bias: extracted core + snakemake orchestration

[cailmdaley]

Closes #224
Closes #226

Big picture

To trust a shear catalogue, we measure the bias of the pipeline that produced it: run the full measurement chain on simulated images with known injected shear, and compare what comes out to what went in. The result is the multiplicative and additive bias (m, c) — the numbers that calibrate, or validate, the survey's shear.

Four roles, three owners:

• Sim campaign (MultiBand_ImSim, external): sets of simulated tiles, identical except for the injected shear — one unsheared reference plus ± pairs in each shear component. Each set — a branch of the campaign — records its injected shear in basic_info.txt.
• Measurement (ShapePipe, CosmoStat/shapepipe#766): detection → PSF → shapes, run identically on every set.
• Orchestration and estimation (this PR): the Snakemake workflow that drives the whole chain, and the estimator that turns matched catalogues into m and c.

The measurement is only as good as the agreement between these components about what was injected, how things are named, and what columns mean. So the design rule throughout: every such fact has exactly one home, and everything else reads it or fails loudly.

The chain

sim campaign (external)
  └─ basic_info.txt — the sims' own record of injected shear
        │ parsed, cross-checked
        ▼
im_manifest → manifest.yaml
  (branch → (g1,g2) map + derived amplitude; the single
   source of truth every downstream stage reads)
        ▼
Snakemake workflow — one container, committed SLURM profile
  ├─ im_pipeline    ShapePipe on each branch × tile
  ├─ im_merge       per-tile catalogues → final_cat_<branch>.hdf5
  ├─ im_extract     comprehensive ngmix catalogue
  ├─ im_calibrate   selection + metacal calibration
  └─ im_mbias       ImageSimMBias → m_bias_results.yaml + provenance

What's here

The campaign manifest. A rule at the head of the DAG parses g_cosmic from every requested branch's basic_info.txt, derives the injected amplitude (cross-checked equal across the sheared branches, sign/axis checked against each branch's name), and writes manifest.yaml into the run root. The estimator takes the amplitude and branch map from the manifest alone; the manifest holds exactly the facts parsed from basic_info.txt.

One estimator. ImageSimMBias pair-matches the ±shear catalogues object-by-object so intrinsic shape noise cancels before averaging (a guard test locks the cancellation), and bootstraps the errors. Estimator variants are config points (pair_match, w_col, global_R_weight) of this one class. It consumes the calibrated shear columns, so the headline quantity is residual m: the bias remaining after the chain's own metacal calibration. Fiducial estimator config: pair-matched, unweighted (#227), with additive-bias subtraction disabled (#226 — on constant-shear sims the subtraction removes the signal itself).

Fail-fast configuration. Science keys (w_col, pair_match, match_radius_deg, n_bootstrap, bootstrap_seed, mask_config) are required — no defaults in code or rule files. Operational keys default visibly in the workflow config.yaml and nowhere else. Run configs are schema-validated; unknown keys are errors. Tiles are given as a tile_ids list. The image-sim mask config is an explicit overlay on the base mask, so the sims-vs-data differences are declared.

One container. A single sp_validation image that can run ShapePipe — that's the one every rule uses.

One run command. snakemake --profile candide … with a committed SLURM profile (partition, node excludes, per-rule resources, OMP_NUM_THREADS=1). Snakemake owns all scheduling: it fans out across branch × tile jobs, and ShapePipe's internal multiprocessing is simply each job's thread count; the whole chain runs MPI-free.

Provenance. Beside each result: every resolved config value, the manifest hash, both repos' commits, the container tag. Derived tracking files are rebuilt each run.

Validation

The current reference measurement (one tile, fiducial campaign, fresh from raw images) is m1 = +0.0197 ± 0.0212, m2 = −0.0158 ± 0.0175, c1 = −0.0027 ± 0.0085, c2 = −0.0032 ± 0.0075 — consistent with zero at the percent level. The seeded bootstrap makes it reproducible, so it doubles as a regression check: refactors should leave it unchanged, and when a change moves it, the diff gets understood rather than waved through. The number itself will evolve as the analysis does. Synthetic-recovery tests (inject known m/c, assert recovery) cover the estimator itself.

Single-tile caveats, stated so the number isn't over-read: tile-to-tile variance is unmeasured at N=1; selection response is not yet accounted; the pair-matched estimator drops the few-percent tail of objects whose detection flips between branches.

Before merge

• Reconciliation run at the companion estimator config point (pair-matched off, weighted) to confirm the one-estimator claim.

Out of scope

Sim-generator changes (tracked as MultiBand_ImSim issues); the multi-tile campaign and selection-response study; cosmology-validation workflow rules (untouched).

— Fable, on behalf of Cail

[cailmdaley]

Status update (PR description rewritten to match): the chain now runs end-to-end from raw SKiLLS images on a single consistent ngmix-v2 grammar vintage, and the m-bias estimator is fixed to a paired, shape-noise-cancelling form with a guard test.

The gate then earned its keep: the first clean run returned m ≈ −1, and the diagnosis landed on two real bugs in the shared calibration path, now filed with full mechanism and evidence:

• Additive-bias subtraction in calibration zeroes the signal on constant-shear sims (m = −1) #226 — the additive-bias subtraction in get_calibrated_m_c zeroes the signal on constant-shear sims (correct for data, fatal for sims; m = −1 exactly, by construction). Three fix options laid out in the issue — this one's a design call on shared calibration code, so it's yours to choose.
• Gatti/DES weights pathological on image sims: N_eff ~1–5% of N, inflates σ(m) ~12× #227 — compute_weights_gatti weights are pathological on the sims (N_eff ~1–5% of N), inflating σ(m) ~12×.

A multiplicative-only hand-diagnostic gives m₁ = +0.14 ± 0.10, m₂ = +0.27 ± 0.11 (grid_1, 1 tile), so the pipeline itself looks healthy. Once the calibration route is chosen, re-running im_calibrate → im_mbias on the existing intermediates is cheap, and we scale to the 5-grid ensemble for the real number. Ball's in your court, Martin.

— Fable, on behalf of Cail

[github-actions]

🔴 ruff found lint / format issues

@cailmdaley — these block the merge into develop. Full list below (also surfaced as annotations in the CI run):

ruff check .

scripts/diagnostics_image_sims.py:16:1: I001 [*] Import block is un-sorted or un-formatted
scripts/diagnostics_image_sims.py:22:1: I001 [*] Import block is un-sorted or un-formatted
Found 2 errors.
[*] 2 fixable with the `--fix` option.

ruff format --check .

Would reformat: scripts/diagnostics_image_sims.py
1 file would be reformatted, 230 files already formatted

---

CI run · Updates on every push and turns green when ruff passes — nothing else to do.