Image simulations workflow#766
Open
martinkilbinger wants to merge 42 commits into
Open
Conversation
added 25 commits
May 17, 2026 19:39
Closed
6 tasks
added 2 commits
June 30, 2026 17:26
…preview # Conflicts: # example/cfis/config_tile_Sx_nomask.ini # example/cfis_simu/readme.txt # scripts/sh/job_sp_canfar_v2.0.bash # src/shapepipe/modules/ngmix_package/ngmix.py # src/shapepipe/modules/ngmix_runner.py # uv.lock
Keep only genuine sims-enablement; revert shared-module changes that were side-effects of the stale ngmix_v2.0 base and collide with in-flight work: - make_cat: restore HSM_G*_PSF grammar (fix/ngmix-psf-columns) + spread_model (714847c, split-761); drops the input_file_List NameError - vignetmaker: drop broad try/except churn (_w_log AttributeError -> test fail) - mask: restore CDSCLIENT_PATH knob (live in 4 example configs) - restore pipeline_tutorial.md (toc.rst ref -> Sphinx build) + run_scratch_local.sh Verification branch only; not for push. Grounds the #766 review.
… -> sp_validation The image-sims workflow splits by concern along the repo dependency arrow (shapepipe produces catalogues; sp_validation consumes + validates them): - ShapePipe (this PR): only the pipeline-module changes needed to run on background-free sims -- the fake_psf module, ngmix BKG_SUB path, numeric sim-exposure-ID handling (exp_utils/sextractor/merge_headers), the cfis_image_sims module configs, create_final_cat's image_sims mode, and make_data_pujol. - sp_validation (separate PR): the snakemake orchestration + the merge/calibrate/m-bias half, rebuilt on sp_validation's native container workflow. The orchestration files removed here are preserved in git history (origin/im_sims) and move there. Incidental stale-base edits to shared modules/standard configs/tests reverted to ngmix_v2.0 (which already carries the fixes: 662fad4 HSM g-cols, 714847c spread_model). Retargeted onto ngmix_v2.0; uv.lock regenerated.
Contributor
|
@martinkilbinger I split off the snakemake parts to sp_validation and kept only the shapepipe changes here (please see updated description above). |
The ngmix_v2.0 retarget slimmed run_job_sp_canfar_v2.0.bash and job_sp_canfar_v2.0.bash of their image-sims code paths, leaving #766 unable to run the image-sims pipeline at all: run_job had no -t/--type dispatch (so the sp_validation workflow's `-t image_sims` was silently swallowed and it ran standard-DATA mode), and its job-1024 completeness check referenced ngmix_interp_runner, a runner that exists nowhere in the codebase. job_sp_canfar rejected `-r symlink` ("vos only for v2.0"), named the Gie and make_cat configs for the data path only, and still invoked shapepipe_run.py (the console entry point is `shapepipe_run`, no extension, since 797eeff). run_job: restored the canonical image-sims runner (full -t/--type dispatch — cfis_image_sims config dir, symlink retrieve, tile_det=sx, faked Uz weight-uncompress, fake_psf under job 512, job-2048 make_cat, and the correct ngmix_runner check). Made it relocatable per #766's goal: a self-derived $SP_ROOT replaces ~/shapepipe for sourcing functions.sh/job_list_help.bash and resolving config dirs, and job_sp_canfar is invoked by explicit repo path (it is not on the container PATH). run_tile_job now re-anchors cwd to the tile work dir on entry: SP_RUN/SP_CONFIG derive from cwd, and run_exp_job leaves cwd in the last exposure dir, so when run_job is called once for the whole pipeline (-j 4095, as the workflow does) the post-exposure tile jobs would otherwise execute in the stale exposure dir. job_sp_canfar: kept the ngmix_v2.0 base — the #746 NUMBER_LIST mechanism that matches the new-grammar container (the canonical runner still uses the retired shapepipe_run -e flag and would crash) — and grafted the image-sims bits: accept `-r symlink`, config_exp_Gie_$retrieve, config_tile_Mc_$psf (the config present in cfis_image_sims and matching run_job's job-2048 check), the bare `shapepipe_run` entry point, and a self-derived $SP_ROOT. Verified on candide against the post-#761 container: `-t image_sims` selects cfis_image_sims + symlink + sx and runs Git -> faked-Uz -> Fe -> Gie(7/7) -> Sp -> Ma -> Mh_exp, each in its correct work dir with new-grammar shapepipe_run. The two scripts run from the bind-mounted repo, so the container supplies only the Python stack. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019Jd85D8xNZZ43FgesabSnk
The image-sims make_cat config lacked SM_DO_CLASSIFICATION, which the newer make_cat_runner reads unconditionally (NoOptionError -> job 2048 crashed, so the per-tile final_cat was never built). The image-sims pipeline has no spread_model_runner stage (make_cat inputs are sextractor + fake_psf + ngmix only), so there is nothing to classify: set it False. With this, a single tile completes end-to-end (Git -> ... -> ngmix -> make_cat), producing a 2345-object final_cat-233-293.fits against the post-#761 container. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019Jd85D8xNZZ43FgesabSnk
create_final_cat selects this column list from the per-tile make_cat output
into final_cat_{sim}.hdf5; the list was still old-grammar (NGMIX_ELL_*,
NGMIX_Tpsf_*, NGMIX_T_PSFo_*, NGMIX_MOM_FAIL) while the post-#761 make_cat
writes NGMIX_G{1,2}_*, NGMIX_T_PSF_ORIG/RECONV_*, so the merge died on
'NGMIX_Tpsf_2M' is not in list.
Rewrite to the v2 grammar, mirroring example/cfis/final_cat.param minus the
two columns the simulation pipeline does not emit (IMAFLAGS_ISO: no
imaging-flag masking stage; NGMIX_MOM_FAIL: superseded by NGMIX_MCAL_TYPES_FAIL,
which is included). Verified: all 65 columns present in the per-tile cat, merge
-> extract -> calibrate runs clean on 1z2z_grid_1.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01D9atA9VPrg7VMC7oBArny9
Contributor
|
Status update (description rewritten to match): pushed the runner restoration ( The end-to-end m-bias run this enabled surfaced two bugs, but both live downstream in the shared sp_validation calibration path — sp_validation#226 and sp_validation#227 — not in anything here. From the ShapePipe side this PR is, as far as we can tell, complete; the trustworthy m-bias number waits on the calibration design call in #226. — Fable, on behalf of Cail |
With HALO_MAKE and SPIKE_MAKE both False (the image-sims mask config), find_stars never runs, so _CDS_stdout is never set and the unconditional read at the end of make_mask raised AttributeError for every CCD. The flag files were already written, so each mask stage 'failed' with 40 bookkeeping errors and exit 1, relying on the runner's continue-on-error to keep the pipeline alive. Guard the CDS block with hasattr, matching the WeightWatcher/rm blocks below it. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TWURLMoixhuMqVrPVdRmCp
This was referenced Jul 10, 2026
) Add an opt-in SEED_FROM_POSITION knob to the ngmix module (default False). When on, the object loop replaces the tile-level RNG with a per-object RandomState seeded from the object's sky position, and rebuilds the prior from that same RNG, so every stochastic step of the fit is a deterministic function of (ra, dec, ccd). Why: image-sim m-bias uses the Pujol estimator — the same scene under different applied shears, with shear response read from the branch difference of the SAME objects. Metacal's fixnoise adds a counter-noise realisation from an RNG seeded per tile, so an object gets different added noise in different branches (detection order differs) and the noise fails to cancel in the difference, inflating sigma_m. Seeding per object from position makes the same object draw the same added noise AND the same fit guesses in every branch, so both cancel and the m-bias error shrinks. Design and box math are Fabian's issue #796. The prior must be rebuilt per object, not just the RNG: the ngmix guesser draws its initial guess via prior.sample(), which consumes the RNG the prior was CONSTRUCTED with (ngmix 2.4.0 guessers.py / joint_prior.py). A per-object RNG alone would leave the guess drawing from the shared tile stream and break cancellation. The fixnoise counter-noise itself rides on the observation's attached noise image (use_noise_image=True), which prepare_ngmix_weights draws from the per-object RNG — so no fresh draw from the bootstrapper RNG is needed. Deviation from #796: Fabian combines the 3-arcsec position boxes as box_x + box_y, which collides along anti-diagonals — boxes (10,20) and (11,19) both give seed 30, so two distinct objects share a noise stream. This replaces the sum with a Cantor pairing (a bijection on the non-negative integers) after a zig-zag fold that maps signed box indices (Dec can be negative) onto non-negative ones, reduced mod 2**32 into the valid RandomState seed range. The box math is kept exactly as his: floor(ra*3600/3)+(ccd+1), floor(dec*3600/3)+(ccd+2), first-epoch coordinates (coord_list[0] convention), so all epochs of one object share the stream. Off by default => production path is byte-identical to develop (verified: metacal g for all five types matches unmodified develop bit-for-bit; the full fast suite passes, 326 tests). New property tests cover the seed's range, sub-box jitter invariance, adjacent-box distinctness, and the resolved anti-diagonal collision. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01J9qEVRCWEekT5ACNB3rjkq
… run Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UvQvyJTEwYcLCwDEgpWMJ6
compile_results filled nfev_fit with np.nan on failed fits, the only int-like column with a float fallback. With SAVE_BATCH enabled, save_results locks column dtypes from batch 1; if that batch has no failures the column locks to int64, and the first later batch with a failure feeds NaN into it, raising "cannot convert float NaN to integer" on append. Use ngmix's own -1 failed/absent-nfev sentinel instead of NaN, keeping the column int64 across every batch. Fixes #795
Companion to the coverage rule in the Snakefile: scans input weight
maps (subsampled), writes tile_coverage_grid_{num}.yaml with an
exclude list read at DAG-build time. Weight-map location is taken
from the get-images ini of the 1p2z variant. Plus two felt fibers
documenting the blank-tile and thread-exhaustion findings.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ling) into im_sims Resolutions: - ngmix.py: Martin's zero-noise edge-stamp guard kept, generalized to np.all(sig_noise > 0) so it stays valid on the per-pixel bkg_rms path, composed with the #803 per-object rng draws (his branch predated #803). - ngmix_runner.py: both imports (os + SqliteDict). - scripts/image_sims_pipeline/*, docs, auxdir tile lists, init_run .sh->.py: Martin's side wholesale — his live operational tooling on candide. The shapepipe-produces/sp_validation-orchestrates consolidation is deferred to the #766 develop-merge review, not imposed on his working branch. - fake_psf.py / vignetmaker.py auto-merges reviewed: robustness/logging only, no PSF content change. - His .felt debug fibers ride along (grid_4 blank tiles, snakemake OMP thread exhaustion) — both operationally important. Verified: tests/module/test_ngmix.py 22/22 in the runtime env (srun). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UvQvyJTEwYcLCwDEgpWMJ6
The image_sims_pipeline Snakefile invokes all container commands through this wrapper; it strips SLURM/PMI/PMIX/OMPI env vars that crash OpenMPI inside apptainer on SLURM compute nodes. No replacement exists in run_scratch_local.sh or run_job_sp_canfar_v2.0.bash. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part of [Tracking] Image simulations (UNIONS-WL/MultiBand_ImSim#1).
Big picture
To measure the shear bias (m, c) of the ShapePipe chain, we run it end-to-end on simulated images with known injected shear and compare output to input. Each distinct injected-shear value defines a branch — the same sky simulated under a different applied shear. The work splits along the dependency arrow:
Boundary rule: ShapePipe owns its module configs (the
.initree lives here, next to the code whose options it names, so code and config change in the same PR); the workflow selects and parameterizes that tree explicitly. Campaign facts — injected shear, branch naming — live upstream in the workflow's manifest and never appear in this repo.What's here
Sim-specific modules and paths:
fake_psfmodule — per-galaxy PSF stamps from the sims' PSF dictionary, drop-in forpsfex_interp's downstream format. The dictionary path is supplied by the workflow (environment-expanded in the config), not hardcoded.BKG_SUBngmix option — skip background subtraction for background-free sims; composes withngmix_v2.0's vignette and centroid options.SEED_FROM_POSITION, Seed fix based on position for Pujol noise cancellation for image simulations #796) — metacal's counter-noise RNG is seeded from each object's sky position, so the same object gets identical fixnoise across shear branches and the noise cancels in the branch difference (measured ~5–6× reduction in σ_m per tile). The seed formula is pinned by unit tests. Enabled only in the sims ngmix config.exp_utils,sextractor_script,merge_headersWARN_MISSING_EXP) and batch-output dtype fix fornfev_fit([BUG] ngmix crash #795).cfis_image_sims/module configs — the one.initree for sim runs — andcreate_final_cat'simage_simsmode.One runner path:
run_job -t image_simsdrives the full module chain and accepts--config_dir; the workflow passes it explicitly, so the workflow's choice of.initree governs every stage. Orchestration lives in sp_validation; the in-repo Snakemake pipeline has been retired on this branch, with its coverage-scan helper (check_tile_coverage.py) kept underscripts/python/.Parallelism model: the orchestrator fans out per-tile jobs, and ShapePipe uses its own multiprocessing within a job; the sims chain launches no MPI.
Validation
The full chain (raw sim tile → per-tile catalogues →
final_cat_<branch>.hdf5) runs on candide. Downstream, the companion PR's estimator measures the residual m left after ShapePipe's own response calibration; on one fiducial tile it is consistent with zero at the percent level, and that fixed-tile measurement serves as the regression check for changes on both branches.— Fable, on behalf of Cail