CLAUDE.md

ShapePipe

ShapePipe is a galaxy shape-measurement pipeline for weak-lensing cosmology. It runs the full chain from raw survey images to calibrated shear catalogues — object detection, PSF modelling, and shape measurement — and was used to produce the first UNIONS cosmic-shear release.

The project is now entering a substantial rework of the shape-measurement pipeline, with the near-term goal of a tight loop between the pipeline and image simulations for validation and calibration. Development is by a small team; reproducibility and clarity matter more than breadth.

Environment — the container is the source of truth

ShapePipe is not a stand-alone library: it needs system tools (Source Extractor, PSFEx, WeightWatcher), MPI, and a specific scientific-Python stack. The supported way to get all of that is the container.

• Dependencies are declared in pyproject.toml as floor ranges; exact versions live in uv.lock, the reproducible manifest generated by uv (never hand-edited). Renovate keeps the lock current (see renovate.json): uv's resolver owns version selection, so cross-package constraints resolve jointly instead of failing update PRs. Bump a floor only when the code actually requires the newer API. System-level tools live in the Dockerfile.
• Images publish to ghcr.io/cosmostat/shapepipe on every push to an integration branch, in two targets:
  • :<tag> — the dev image: the full stack plus interactive tooling and the test / lint / doc extras.
  • :<tag>-runtime — a slim image for batch jobs and downstream FROM clauses.
• On a cluster, run with Apptainer:

  apptainer build --sandbox shapepipe docker://ghcr.io/cosmostat/shapepipe:develop
  apptainer shell --writable shapepipe
  cd /app && shapepipe_run -c /app/example/config.ini

• For development, work inside the dev image — it carries vim, ripgrep, pytest, and the rest. A common pattern is a long-lived --writable Apptainer sandbox with a host clone of the repo bind-mounted in and pip install -e pointed at it, so edits on the host are live inside the container.

Shadow a library build with PYTHONPATH (no image rebuild). When you need the container to run a different build of a pure-Python library than the one baked into its venv — a feature worktree, an unreleased branch, this repo's own src/ against a stale image — prepend the host checkout to PYTHONPATH. Python resolves the prepended path first, so the on-disk version shadows /app/.venv/... without touching the image. This is the local-testing counterpart of a git-ref dependency (e.g. cs_util @ develop in pyproject.toml): the dep change makes CI build the right version; the shadow lets you test that version now, before any rebuild. The recipe:

apptainer exec --bind /n17data,/automnt <image.sif> bash -c \
  "cd <repo-worktree> && \
   PYTHONPATH=/path/to/libfoo-checkout:<repo-worktree>/src \
   python -m pytest <targets> -o addopts='' -q"

Notes: the checkout path is the parent of the importable package dir (the dir containing foo/, not foo/ itself); list several :-separated to stack shadows; -o addopts='' clears pyproject.toml's pytest defaults when a plugin they reference (e.g. pytest-cov) isn't in the image. Use this for a quick verify; land the real fix as the pyproject.toml / uv.lock dep change so CI and the next image agree.

Testing container changes: build remotely, pull locally. Don't apptainer build images on a cluster — quotas are tight and the build is slow. The loop for any change to Dockerfile / pyproject.toml / uv.lock is: edit → push → let GitHub Actions build and publish to GHCR → apptainer pull docker://ghcr.io/cosmostat/shapepipe:<branch>[-runtime] on the cluster → test. Watch the remote build with gh run watch (or gh run list --branch <branch>). The only things that run locally are the pull and the test. On a quota-limited cluster, keep SIFs and Apptainer's scratch off $HOME: point APPTAINER_TMPDIR / APPTAINER_CACHEDIR at a roomy data partition and pull SIFs there.

Full detail: docs/source/installation.md and docs/source/container.md.

Layout

• src/shapepipe/ — the package (src-layout). modules/ holds the pipeline modules and their *_runner.py wrappers; pipeline/ is execution and file I/O; utilities/; canfar/ is CANFAR/cluster job orchestration. Console entry points (shapepipe_run, summary_run, canfar_*) are defined under [project.scripts].
• tests/ — the whole test suite, one discovery root: module/ (per-module unit/property/integration tests), unit/ (structural), science/ (fast guardrails), cluster/ (candide-only), helpers/ (shared library code). See tests/README.md.
• example/ — a runnable example pipeline (example/config.ini) on a single CFIS tile; doubles as the CI smoke test.
• scripts/ — shell / Python / notebook helpers (sh/, python/, jupyter/), symlinked onto $PATH inside the image.
• docs/ — Sphinx sources; the API docs are generated from docstrings.

Development workflow

• develop is the integration branch — open PRs against it. main / master are release branches.
• Tests run with pytest. CI runs them inside the dev image, so the suite exercises exactly what ships — run them the same way, in the dev container.
• CI (.github/workflows/deploy-image.yml): every PR and push builds the image and runs the test suite + the example pipeline + binary smokes; pushes to develop / main / master additionally publish the images. API docs deploy from master (cd.yml).
• Style: PEP 8; numpydoc docstrings on public modules, classes, and methods. Match the surrounding code.

Project knowledge

Durable project knowledge — decisions, architecture, plans — lives in this repo's pull requests, issues, docstrings, and docs/, alongside notes some maintainers keep in their own stores outside it. A .felt/ directory (a markdown "fiber" note store used with the felt CLI) is not tracked here: it's gitignored, and where it exists it's a machine-local symlink into a private, separately git-synced store, so a fresh clone won't have one. Record durable decisions in the PR, issue, or docs where the change lives.