Rift o4d junior calmarg in loop: AFTER main 'distance' merge#139
Merged
oshaughn merged 297 commits intoJun 30, 2026
Merged
Conversation
…kes_digits Update util_TestSpokesIO.py: fix digits call to opt.digits
…rough (incl. time samples) Adds `make pp` (pp-build + pp-validate): exercises the FULL top-level builder util_RIFT_pseudo_pipe.py -> helper_LDG_Events.py -> args generation -> create_event_*, not just the lower-level builder that dag-build calls directly. Offline build-validate (the established pattern: .travis/test-build.sh + demo/pipeline/zero_spin_phenomD), reusing the zero-spin IMRPhenomD ini + ref coinc + a placeholder fake-data cache, so no GPU/data run is needed for the threading check. Zero spin (--assume-nospin); iterations forced small (--internal-force-iterations). Confirms EVERYTHING threads through the generated pipeline (validated, not just emitted): - calmarg --calibration-envelope-directory / --calibration-n-realizations / --calibration-fused-kernel land in args_ile.txt, ILE.sub AND ILE_extr.sub; - TIME SAMPLES: --add-extrinsic-time-resampling + --internal-ile-srate-time-resampling -> --srate-resample-time-marginalization 4096 in the wide AND extrinsic (ILE_extr) stages, alongside --time-marginalization; - zero-spin IMRPhenomD; small iteration count; full top-level DAG produced. (Note: --last-iteration-extrinsic-time-resampling is a transient builder arg consumed at build time -- its persistent effect is the --srate-resample-time-marginalization in ILE_extr.sub, which is what we assert.) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…a (breadcrumb) Primes and launches a full util_RIFT_pseudo_pipe.py pipeline that ACTUALLY RUNS on the zero-noise CI fake data (not just the offline pp-build threading check). Pieces: - util_SimInspiralToCoinc.py makes a real coinc from the injection (H1/L1/V1, t=1000000014, m1=35/m2=30, SNR 17.5) -- `make pp-coinc`/ci_coinc.xml (regenerated, not committed). - calmarg_ci.ini: CI-matched (FAKE-STRAIN channels x3, srate 4096, seglen 8 -> the segment [1000000008.236,1000000016.236] sits inside the cache's 328s frame, fmin 10, zero spin, mc [23,35]). - pp-run-build runs pseudo_pipe with the real zero_noise.cache + CI PSD + calmarg (--calmarg-* --calmarg-fused-kernel) + time-resampling (--add-extrinsic-time-resampling + --internal-ile-srate-time-resampling 4096) + --assume-nospin + small forced iterations, and asserts the RUNNABLE bits threaded (real cache, FAKE-STRAIN, event time, and that ILE_extr.sub carries calmarg + --srate-resample-time-marginalization). - pp-run builds + condor_submit_dag. Verified: builds clean, submits (the extrinsic stage produces TIME SAMPLES with calmarg on). Single GPU on cardassia (CUDA_VISIBLE_DEVICES=0). This is the runnable counterpart to pp-build; leaves a working end-to-end breadcrumb for exercising the full pipeline + calmarg + time samples later. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…TY_RIFT_IMAGE is set
CIT is container-only (needs SINGULARITY_RIFT_IMAGE + --use-osg). Make pp-run "just work"
there: exporting SINGULARITY_RIFT_IMAGE (or passing OSG=1) auto-populates the OSG target
space --
--use-osg (-> builder adds --use-singularity, reads SINGULARITY_RIFT_IMAGE),
--use-osg-cip, --use-osg-file-transfer,
--ile-additional-files-to-transfer cal_env/{H1,L1,V1}.txt + the PSD (no shared FS on OSG),
and links the CI frames into rundir_pp_run/frames_dir for --use-osg-file-transfer.
A guard errors if OSG=1 but SINGULARITY_RIFT_IMAGE is unset. On cardassia (env unset,
OSG=0) it is a clean no-op -- the local shared-FS pp-run is unchanged.
CIT: export SINGULARITY_RIFT_IMAGE=/cvmfs/.../rift.sif ; make pp-run
Verified by dry-run (make -n): env set -> all OSG flags + transfer list + frames link;
env unset -> zero OSG flags; OSG=1 w/o env -> guard fires. NOTE: full OSG validation
(esp. transferred-file path rewriting for the PSD/cal envelopes) awaits a working CIT
container build (currently blocked); the flag layer follows the standard RIFT OSG pattern.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Two small CIT fixes (per review): 1. pp-run-build wrongly FAILed "not the real CI cache" in OSG mode: OSG uses local.cache (built on the remote worker), not zero_noise.cache. The cache-name assertion is now OSG-aware -- OSG=1 just notes local.cache (no hard fail); local shared-FS still asserts zero_noise.cache. The other runnable-bit checks (FAKE-STRAIN, event time, ILE_extr calmarg + time-resampling) are unchanged. 2. pp-run now prints a warning after submit if LIGO_USER_NAME or LIGO_ACCOUNTING (a.k.a. LIGO_ACCOUNTING_GROUP) are unset -- at CIT, without these the condor jobs never queue. Silent when set. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…headroom)
Per review: a container needs ~4 GB more disk than the MB-scale shared-FS baseline (the
.sif image is transferred + unpacked on the worker). OSG mode now adds
--internal-{ile,cip,general}-request-disk (default PP_OSG_DISK=4G, overridable) to the
pp-run pseudo_pipe invocation, covering all container job types. Local (non-OSG) mode
unchanged. Verified by dry-run: OSG=1 -> all three disk flags at 4G; OSG=0 -> none.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…disk request
The container +4GB disk bump only took for cip/general; ILE stayed at 4M because
calmarg_ci.ini set internal-ile-request-disk="4M" in [rift-pseudo-pipe], and an ini value
there OVERRIDES the util_RIFT_pseudo_pipe.py CLI (cip/general weren't in the ini, so they
got the CLI 4G).
Fix: remove the disk line from calmarg_ci.ini and drive ALL THREE disk requests from the
CLI, ALWAYS: --internal-{ile,cip,general}-request-disk = PP_DISK, where PP_DISK is 4M
(local baseline) or 4G (container/OSG). Verified by dry-run: OSG -> all three 4G; local
-> all three 4M; ini sets no request-disk.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…senames, present)
On OSG (no shared FS) the wide-ILE PSD and cal-envelope args referenced ABSOLUTE paths the
worker can't see, and the per-IFO PSD files the transfer list expected were never created.
PSD (demo Makefile): on OSG, drop --use-online-psd-file (which forced the absolute HLV
path into the args) so RIFT emits the normal per-IFO basenames --psd-file H1=H1-psd.xml.gz
(which it lists + transfers), and COPY the CI PSD into the run dir as {H1,L1,V1}-psd.xml.gz
so those files exist AND carry the real PSD. Local (shared-FS) mode keeps
--use-online-psd-file (absolute path, no transfer). Verified: OSG args are basenames,
files present; local unchanged.
Cal envelopes (util_RIFT_pseudo_pipe.py, the general fix -- part of task oshaughn#23): when
--use-osg-file-transfer, reference --calibration-envelope-directory as '.' (the per-IFO
<IFO>.txt land flat in the job scratch dir) and auto-append them to the ILE transfer list,
so calmarg-on-OSG works without the user remembering --ile-additional-files-to-transfer.
Local mode unchanged (absolute path). Verified the OSG args_ile.txt + helper_transfer_files.txt.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
condor refuses "Transfer of symlinks to directories is not supported", so the OSG frames_dir is now `cp -r` of the CI frames (not ln -sfn). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…d to ILE as stray arg) Hardening (task oshaughn#26): args_ile.txt starts with a placeholder token 'X' (stripped by create_event_parameter_pipeline_BasicIteration), not the exe. util_CalPilotStage only recognized an exe first-token, so 'X' leaked into the pilot's ILE dump command as a stray positional argument. It happened to be tolerated by optparse, but is wrong/fragile. Now drop the first token when it is 'X' or contains 'integrate_likelihood'. Verified on a real pseudo_pipe args_ile.txt: X dropped, $(macro) stripped, pilot opts removed for re-supply, calmarg + time-marginalization kept. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…pipe (CI data)
Adds PP_PILOT toggle + `make pp-run-pilot` (and pp-run-pilot-build): pp-run with
--calmarg-pilot, so the full top-level pilot DAG (harvest->dump->fit->consolidate->seed
wide_{N+1}) runs on the CI fake data. build-validate asserts CALPILOT.sub runs
util_CalPilotStage.py, the CALPILOT job is in the DAG, and the wide ILE args carry the
--calibration-proposal-breadcrumb seed. Honours OSG/CIT like pp-run. Verified the build.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…(task oshaughn#23) The CALPILOT job runs ILE internally, so on OSG it needs the same container + input set as a wide ILE job. write_calpilot_sub gains use_osg/use_singularity/frames_dir/transfer_files (mirrors write_ILE_sub_simple): - runs in the singularity image (exe at SINGULARITY_BASE_EXE_DIR; transfer_executable False; MY.SingularityImage/BindCVMFS/flock_local; HAS_SINGULARITY requirement); - a calpilot_pre.sh prescript rebuilds local.cache (relative paths) from the transferred frames, then execs the stage; - transfer_input_files = transfer_files (PSD + cal envelopes, sans the wide grid) + frames_dir + composite + args_ile.txt; transfer_output_files = the consolidated breadcrumb; stage args reference BASENAMES (no shared FS), workdir '.'. - refinement (--prev-breadcrumb) is skipped on OSG (the prev breadcrumb is produced at runtime, can't be reliably listed for transfer at iteration 0) -> each OSG pilot is an independent cold start, which the prior-shrinkage fit makes safe. create_event_parameter_pipeline_BasicIteration passes the OSG params + transfer_file_names to the calpilot job. ILE robustness: the wide-ILE breadcrumb seed load is now wrapped in try/except -> a missing/partial/invalid breadcrumb (esp. under OSG file transfer) falls back to PRIOR cal draws with a warning instead of killing the job. DONE: the CALPILOT jobs RUN on OSG and produce cal_consolidated_N.npz (transferred back). REMAINING (task oshaughn#23): consuming the seed on OSG -- transferring cal_consolidated_{N-1}.npz to the wide_{N+1} ILE jobs (pseudo_pipe basename ref + the iteration-start-absent edge), so the wide jobs use the learned proposal rather than always falling back to prior. UNTESTED off-CIT: validate the container + transfer on a real OSG run. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…ist (task oshaughn#23 complete) Wide-ILE seed consumption on OSG (util_RIFT_pseudo_pipe.py): when --use-osg-file-transfer, reference the proposal breadcrumb by BASENAME (cal_consolidated_$(macroiterationprev).npz), add it to the ILE transfer list, and create a placeholder cal_consolidated_-1.npz so condor's transfer for the first iteration (prev=-1, never produced) succeeds -- ILE's breadcrumb load is try/except and falls back to the prior for the placeholder. So wide_{N+1} now actually consumes the learned proposal on OSG, not just falls back to prior. Clean CALPILOT transfer list: write_ILE_sub_simple mutates transfer_file_names in place (appends frames_dir, ile_pre.sh, the grid), and CALPILOT is built after the wide ILE, so it had inherited that pollution (frames_dir x3, the wide grid, the ILE prescript). Snapshot a clean PSD+cal-envelope transfer list BEFORE those mutations and pass it to the pilot. Verified: the CALPILOT transfer_input_files now lists each file exactly once (PSD, cal envelopes, composite, args_ile.txt, frames_dir, calpilot_pre.sh, prev breadcrumb). This completes the OSG pilot file transfer (CALPILOT runs in-container + transfers I/O; wide_{N+1} gets the seed). UNTESTED off-CIT -- validate on a real OSG run. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
… rundir_pp_run pp-run-build starts with `rm -rf $(PP_RUN_REAL)`, and pp-run-pilot reused rundir_pp_run -- so launching the pilot demo DESTROYED an in-progress vanilla pp-run. pp-run-pilot[-build] now overrides PP_RUN_REAL=rundir_pp_pilot so the two run directories are independent and neither clobbers the other. clean removes both. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…n the OSG prescript too On OSG the CALPILOT.sub executable is calpilot_pre.sh (the prescript that rebuilds local.cache then runs the container's util_CalPilotStage.py), so the stage name is in the prescript, not CALPILOT.sub. The build-validate grep now checks BOTH CALPILOT.sub and calpilot_pre.sh (grep -qs), fixing a spurious "CALPILOT.sub does not run util_CalPilotStage.py" on OSG. Pipeline-writer/demo-level only -- no container rebuild. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…+ PoC
The decade-old "save the extrinsic distribution to inform the next iteration" goal,
generalized from the cal pilot's breadcrumb. GMM-first (mcsamplerEnsemble is already
seedable via gmm_dict).
- breadcrumbs.py (schema v2): the extrinsic slot now carries a per-param-group Gaussian
mixture -- means/covariances/weights/bounds + the param NAMES (so dim-group indices
reconstruct against the next run's params_ordered). cal + extrinsic coexist in one
breadcrumb. save/load round-trip test (cal + extrinsic) PASSES.
- RIFT/calmarg/extrinsic_handoff.py:
fit_extrinsic_proposal(samples, log_weights, groups, bounds, n_comp) -- per group, fits
with RIFT's OWN gaussian_mixture_model.gmm (the exact fitter the sampler uses in
update_sampling_prior), so stored means/covs are in the model's internal frame and
restore byte-identical -- no coordinate guesswork, no sklearn.
gmm_dict_from_breadcrumb(extrinsic, params_ordered) -- reconstructs gmm objects keyed by
dim-group indices (looked up by name), ready to seed mcsamplerEnsemble's gmm_dict.
Standard groups (ra,dec),(distance,incl),(phi_orb,psi). Handles the GMM running on cupy.
- PoC (__main__): synthetic BIMODAL sky posterior -> fit -> breadcrumb -> load -> seed ->
the seeded sky GMM recovers BOTH modes. PASS.
Worktree branch rift_O4d_junior_extrinsic_handoff (off the calmarg branch); does not touch
the running pipeline checkout.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…cept) Complete the GMM extrinsic-handoff loop: - ILE (integrate_likelihood_extrinsic_batchmode, EXECUTE-POINT -- needs container rebuild): --extrinsic-proposal-output harvests the run's extrinsic posterior samples + importance weights from sampler._rvs after integrate (same weight recipe as the distance-grid export, incl. the GMM sampler's raw-integrand storage), fits per-group GMMs via RIFT.calmarg.extrinsic_handoff, and writes a breadcrumb. --extrinsic-proposal-breadcrumb seed side (pre-fill gmm_dict) was added prior. Both wrapped in try/except so the handoff can never break a production integration. - DESIGN_extrinsic_handoff.md: the decade-old "carry the extrinsic posterior between iterations" goal, GMM-first rationale (mcsamplerEnsemble.gmm_dict is trivially seedable), module/ILE pieces, PoC result, pilot-DAG plug-in plan, and the AV partial-reset limitation (task oshaughn#30: AV resets every integrate(), can only contract). PoC (python -m RIFT.calmarg.extrinsic_handoff) and breadcrumb round-trip (python -m RIFT.calmarg.breadcrumbs) both PASS. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Make the extrinsic handoff usable end-to-end (standalone; does NOT require the cal
pilot), gated by --extrinsic-handoff and requiring the GMM sampler:
- util_RIFT_pseudo_pipe.py --extrinsic-handoff: thread per-event
--extrinsic-proposal-output extr_proposal_$(macroiteration)_$(macroevent).npz and the
seed --extrinsic-proposal-breadcrumb .../extr_consolidated_$(macroiterationprev).npz
into args_ile.txt (OSG: basename + transfer-list + iteration-0 placeholder; shared FS:
absolute path), mirroring the cal breadcrumb. Warns if --ile-sampler-method != GMM.
Passes --extrinsic-handoff[-select] through to the pipeline builder.
- util_ExtrinsicConsolidate.py (NEW): pick the single most representative per-event
proposal (default by lnL = nearest the peak; neff/n_samples also available) ->
extr_consolidated_<it>.npz. Skips unreadable/placeholder inputs; ALWAYS writes output
(empty if nothing valid) so the next iteration's seed/transfer never fails.
- dag_utils_generic.write_extrconsolidate_sub (NEW): the consolidation job, LOCAL universe
on the submit node (pure-python file selection, no GPU/ILE/container/frames). On OSG the
per-event ILE outputs are transferred back to <wd>/iteration_<it>_ile, so it reads them
from the shared FS -- no per-event input transfer (which condor cannot glob).
- create_event_parameter_pipeline_BasicIteration: one consolidation node per iteration,
gated behind that iteration's unify (ILE barrier), and the next iteration's wide ILE jobs
depend on it: unify_{it} -> EXTRCONSOLIDATE_{it} -> wide ILE_{it+1}.
- ILE save side: record true lnL + neff in the proposal breadcrumb meta so consolidation
can pick the most representative point.
- demo/rift/calmarg: `make extr-build` builds + offline-validates the whole thread
(args_ile.txt flags, EXTRCONSOLIDATE.sub, unify->consolidate->next-ILE DAG edges);
separate rundir_pp_extr so it never touches other run dirs.
Verified: `make extr-build` passes; util_ExtrinsicConsolidate standalone tests pass
(picks highest-lnL, skips placeholders, writes empty on no-input).
NOTE: the ILE binary change (--extrinsic-proposal-output/-breadcrumb, save+seed) is
EXECUTE-POINT -- rebuild the container before an OSG/CIT run. The convergence subdag
(--first-iteration-jumpstart) does not yet carry --extrinsic-handoff (same as --calmarg-pilot).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…rget Found by running the GMM extrinsic handoff on a real GPU (cardassia, NVS 510); the full loop now works end-to-end (iteration-0 writes proposal -> consolidate -> iteration-1 prints "Extrinsic GMM SEEDED ... [(4,5),(3,2),(0,1)]" for all three groups -> integrates -> writes the next proposal): - reconstruct_gmm: move self.bounds onto the GPU (identity_convert_togpu). The sampler's score()/_normalize write into a cupy array, so a leftover numpy bounds raised "non-scalar numpy.ndarray cannot be used for fill". - gmm_dict_from_breadcrumb(existing_keys=...): match each breadcrumb group to the sampler's actual gmm_dict key by dim-SET and permute the stored means/covariances/bounds columns into that key's order. Fixes the phase/pol group being silently dropped because the sampler pairs (psi,phi_orb)=(0,1) while the breadcrumb stored (phi_orb,psi)=(1,0). - reconstruct_gmm(cov_inflate=2.0): broaden the seed (a warm start should be conservative; the ensemble sampler can contract but starves if seeded too tight). Mitigates -- does not rescue -- a degenerate source: on a bad batch the sampler _reset()s gmm_dict[k]=None, i.e. discards the seed and continues cold (correct safety net). So seed quality tracks the SOURCE iteration's convergence; a useful (accelerating) seed needs n_eff in the hundreds, i.e. a real --n-max / larger GPU, not the tiny smoke (n_eff~1 -> seed safely discarded). - demo/rift/calmarg: `make extr-run[-build]` -- tiny GMM extrinsic-handoff pipeline on the CI data (300 initial / 200 per-gen intrinsic, 50 evals/ILE job, n-chunk 4000, n-max bounded to 40000 vs the 4,000,000 production default, >=2 iterations). Derives a run-specific ini (sed) because [rift-pseudo-pipe] ini values override the CLI. Separate rundir_pp_extr_run. DESIGN_extrinsic_handoff.md: documents the GPU validation, the two bugs, and the seed-quality-vs-source-convergence finding. ILE binary change is EXECUTE-POINT (container rebuild for OSG/CIT). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Background
----------
util_ConstructIntrinsicPosterior_GenericCoordinates.py has long used
three CLI flags for declaring how a parameter is treated:
--parameter X both fit dim AND MC sampling dim
--parameter-implied X fit dim only (the converter produces X from the
data file's columns; the MC integrator never
sees it)
--parameter-nofit X MC sampling dim only (the integrator integrates
over it; the fit never sees it)
util_ConstructEOSPosterior.py declared the same three flags but never
honoured them: the integrator at line 487 hardcoded
`low_level_coord_names=dat_orig_names` in its convert_coords closure,
which only worked when the sampling basis equalled the data-file basis;
sampler.add_parameter iterated over coord_names (the fit basis); the
arity dispatch for likelihood_function keyed on len(coord_names); and
sampler.integrate was passed *coord_names rather than
*low_level_coord_names. The net effect: any user who tried to fit in a
transformed basis (e.g. via the new --supplementary-coordinate-code
plugin) silently got a wrong likelihood evaluation -- the rotation was
applied an extra time inside convert_coords every Monte Carlo step.
What this commit changes
------------------------
bin/util_ConstructEOSPosterior.py
* Parameter-resolution block rewritten to mirror IntrinsicPosterior's
semantics, plus a clean fallback to dat_orig_names when none of the
three flags are supplied (legacy bare-invocation unchanged). Seven
CLI permutations now map to documented (coord_names,
low_level_coord_names) pairs.
* The convert_coords closure used by the integrator captures
low_level_coord_names as its input basis (was dat_orig_names). The
initial dat->X conversion still uses dat_orig_names, since that's
the basis of the file columns.
* Sampler add_parameter loop now iterates over low_level_coord_names
(the MC basis), and sampler.integrate is passed *low_level_coord_names.
* The arity-dispatched likelihood_function definitions key on
len(low_level_coord_names) and route every input -- including the
scalar branches -- through convert_coords so a non-trivial converter
is never silently bypassed.
* Output-writer iterates samples by low_level_coord_names (the keys
sampler._rvs actually carries) and applies the "constant fill"
check in the sampling basis, not the fit basis. Implied (fit-only)
coords correctly skip the output file.
* Added a guard: if low_level_coord_names != coord_names but no
coordinate plugin is supplied, raise a clear error instead of
silently feeding samples through an identity convert_coords into a
fit built in a different basis.
* Help text for --parameter / --parameter-implied / --parameter-nofit
rewritten to describe what each flag actually does now.
RIFT/hyperpipe/coords.py
* HyperCoordSpec.from_strings accepts integration ranges for names in
coords-nofit (the MC sampling basis is coords-fit + coords-nofit);
unknown range names are still rejected.
* HyperCoordSpec.validate accepts empty coords-fit so long as
coords-implied covers the fit basis and coords-nofit covers the
sampling basis; emits distinct errors for empty-fit vs empty-sample.
* to_parameter_args emits --integration-parameter-range for the
sampling basis (parameters + nofit), not just parameters.
* to_puff_args and to_test_args emit --parameter for the sampling
basis -- the puff lane and convergence-test driver operate on the
data-file columns, which is the sampling basis after decoupling.
RIFT/hyperpipe/config.py
* validate_config accepts empty coords-fit when coords-implied
(fit-side) and coords-nofit (sample-side) are non-empty.
demo/hyperpipe/hyperpipe_conf_linear_uvw.yaml
* Rewritten to actually exercise the decoupled path: coords-implied
"u v w" (fit), coords-nofit "x y z" (sample), coords-sample ranges
in (x, y, z), coord-module pointing at the linear plugin with the
uvw_rotated chart. Iteration / puff / marg stay in (x, y, z); the
EOS posterior fits in (u, v, w) and writes its posterior in
(x, y, z).
Verified
--------
* Parameter-resolution unit test (in this commit's worktree) covers 7
CLI permutations -- legacy no-flags, legacy --parameter, IntrinsicPosterior
--parameter+implied and --parameter+nofit, the new --implied-only,
--implied+nofit, and full --parameter+implied+nofit -- all map to
the documented (coord_names, low_level_coord_names) pairs.
* HyperCoordSpec unit test covers the new decoupled emit (post sees
implied/nofit and ranges; puff/test see the sampling basis only),
a legacy-regression case (unchanged output), the two new validation
errors (empty fit, empty sample), the new "range for nofit name"
permission, and the still-rejected "unknown range name" case.
* AST + yaml parses on every edited file.
* validate_config passes on hyperpipe_conf_linear_uvw.yaml plus the
demo's baseline and tracer yamls.
… GPU Attempting the seed-acceleration demo on the CI point (SNR~17.5, lnLmax~90-115) showed the ensemble (GMM) sampler does not converge there: n_eff pinned at ~1 through ~200k samples, with OR without calmarg (vanilla GMM: 1.00007 at 196k / 50 iterations). GMM collapses onto the dominant sample at a sharp high-SNR peak; AV (the production sampler) handles these but is not seedable. So the GMM->GMM handoff is correct+safe but cannot bootstrap a useful seed on real high-SNR data -- its payoff is gated on seedable/partial-reset AV (task oshaughn#30/oshaughn#25) or a cross-sampler AV->GMM seed (fit_extrinsic_proposal already accepts any sampler's samples). Recorded in DESIGN_extrinsic_handoff.md. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…sal-adapt) + cross-sampler findings - --extrinsic-proposal-adapt (default OFF = freeze): the seeded GMM groups are no longer re-fit each iteration. Re-fitting a seed on a bad first batch dies in the GMM init (random.choice "probabilities are not non-negative") and triggers _reset, discarding the seed. _train already skips groups with gmm_adapt=False, so freezing preserves the seed. Freezing is also the right semantics for a handed-off / cross-sampler proposal. With freeze the seeded run completes with 0 resets and n_eff rises from cold ~1 to ~5-10. - DESIGN_extrinsic_handoff.md: document the cross-sampler AV->GMM result. AV converges as a source (n_eff~7 at 400k, lnLmax~143); the frozen seed lands cleanly and lifts n_eff, but the seeded GMM INTEGRAL is wrong (sqrt(2 lnLmax)=nan, Z~1e-4 vs cold ~1e43) -- the proposal is importance-sampling a displaced region. Two suspects to audit (no more blind GPU): AV-vs-GMM _rvs coordinate convention (angle vs cosine for incl/dec), and cov_inflate pushing distance out of [1,1000] into NaN likelihood. Same-sampler GMM->GMM round-trips cleanly. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…d weights, ESS n_comp, distmarg) Debugging the wrong-integral the GPU run showed (the seeded GMM integrated as if lnL~44 vs the true ~140), found+fixed four real issues; the cross-sampler seed is now numerically correct (finite lnLmax, valid Z) end-to-end: 1. SAVE side used the sampler's stored log_weights, but mcsamplerGPU/AV stores log_weights = tempering_exp*lnL + ln(prior) - ln(s_prior) (adapt-weight-exponent baked in). Fitting the GMM to those flattened weights displaces the proposal. Now build the TRUE untempered weight from log_integrand + log_joint_prior - log_joint_s_prior and prefer the raw components over 'log_weights'. (GMM's own _rvs is untempered -> GMM->GMM unaffected.) This took the seeded n_eff from ~5 to ~26. 2. cov_inflate default 2.0 -> 1.0: a frozen seed should match the source, not be widened (inflation pushes samples past hard bounds -> NaN likelihood). 3. fit_extrinsic_proposal: cap mixture components by the weight ESS (k <= ESS/(d+2)) and DROP any non-finite component (renormalize; skip group if none survive). A starved source collapses a component to a singular/NaN covariance that poisons the whole seed. 4. The persistent nan lnLmax was distance sampled against [1,1000]: a seeded distance Gaussian spills past the bound -> NaN. The calmarg path is meant to run with --distance-marginalization (the fused kernel IS a distmarg kernel); with distmarg on, the seeded integral is finite and valid. (Gap: pseudo_pipe/extr-run don't add --distance-marginalization yet -- noted in doc.) Result (distmarg on, CI point SNR~17.5): seeded GMM has 0 resets, finite lnLmax, valid Z, but n_eff ~1 == cold n_eff ~1. The handoff is correct+safe but does NOT accelerate here because GMM does not converge on this peak (cold or seeded) and the AV source (n_eff~5) is too under-converged to inform a strong seed. Hard evidence the payoff needs seedable AV (task oshaughn#30) or a converged source. Full analysis in DESIGN_extrinsic_handoff.md. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…nstance (XML compat) copy_lsctables_sim_inspiral iterated lsctables.SimInspiralTable.validcolumns.keys() (the full schema) and did bare getattr(row, simattr) for string columns (waveform/source/numrel_data/ taper) and numeric columns. On the current igwn_ligolw + lalsuite stack, ILE-written sim_inspiral tables contain only the columns actually set, so the schema view and the written columns drift apart -> reading a saved ILE output_*.xml.gz raised "AttributeError: 'SimInspiral' object has no attribute 'waveform'" (and would equally fail on any absent numeric column via the else branch). Fix: skip columns not present on the row instance (hasattr guard), after the process_id/simulation_id default-setting branch (which doesn't read the row). RIFT's own grids (written via lsctables.New with all columns) are unaffected; column-subset tables now round-trip. Verified both a 300-row RIFT grid and a waveform-less ILE-style table read with no AttributeError. Per /home/oshaughn/BREADCRUMB_rift_xml_compat.md. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
… for the calmarg pipeline There was no pipeline-code gap: --internal-marginalize-distance already composes cleanly with --calmarg-fused-kernel (verified -- args_ile gets --distance-marginalization + a util_InitMargTable lookup table AND --calibration-fused-kernel), and the fused kernel does NOT require distmarg (it has both Q_fused_calmarg_cupy and Q_fused_calmarg_distmarg_cupy; the ILE binary wires whichever applies). The only gap was that the demo targets didn't expose distmarg. - demo Makefile: PP_DMARG toggle (default 0, optional) -> --internal-marginalize-distance --internal-distance-max PP_DMAX, threaded into extr-build, extr-run-build, pp-run-build. (Distinct from the direct-ILE dag-build DMARG knob, which uses a pre-built lookup table.) extr-validate checks --distance-marginalization + lookup table when PP_DMARG=1. Verified `make extr-build PP_DMARG=1` passes. - DESIGN_extrinsic_handoff.md: corrected -- distmarg is OPTIONAL with the fused kernel, not required; RECOMMENDED with --extrinsic-handoff (removes distance + its hard bound from the seeded GMM proposal, which was the source of the boundary-NaN). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…fairdraw cupy crash)
BREADCRUMB_export_cal_posterior.md: the final export with extrinsics did not carry the
recovered calibration posterior. Now --calibration-export-posterior (ILE) /
--calmarg-export-posterior (pseudo_pipe): at the fairdraw export, for each fair-draw sample
draw ONE cal realization in proportion to its posterior weight (per-realization likelihood
components from return_cal_components, times the importance weight cal_log_weights) and write
a SELF-CONTAINED sibling <output>_<event>_cal.dat with the FULL draw -- intrinsic + extrinsic
+ the drawn realization's spline nodes as labeled cal_<IFO>_amp_<k>/cal_<IFO>_phase_<k>
columns. (The fairdraw LIGOLW/.dat schema can't carry arbitrary columns, so per the user the
cal posterior rides a row-aligned sibling .dat with the whole draw, plottable as-is.)
- node retention: the production prior path now keeps the cal node vectors
(draw_prior_realizations_with_nodes) when the flag is set; the seed path already returns them.
- verified on GPU: writes 1 sample x 90 cols incl 60 cal cols (amp_0..9 + phase_0..9 over H1,L1,V1).
Also fix a PRE-EXISTING crash this surfaced: mcsamplerEnsemble (GMM sampler) fairdraw on GPU
did `self.xpy.min([n_extr, 1.5*eff_samp, 1.5*neff])` -- cupy.min has no Python-list overload
("'list' object has no attribute 'min'"), so ANY GMM-sampler fairdraw export on GPU crashed
(independent of calmarg). Use Python min() of floats.
ILE binary is EXECUTE-POINT (container rebuild to run on OSG/CIT).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…+ thread cal-export into demo PILOT OSG bug: iteration 0 seeds from cal_consolidated_$(macroiterationprev).npz with macroiterationprev=-1 -> cal_consolidated_-1.npz, the 0-byte placeholder pseudo_pipe creates so condor's transfer_input_files of that path does not fail. Locally the file simply does not exist (the "missing -> fall back to PRIOR" check fires); on OSG it IS transferred in, so it EXISTS but is empty, and np.load raised "EOFError: No data left in file", crashing the first-iteration ILE. Fix: treat a missing OR EMPTY breadcrumb as "not present yet" (size guard before any load), for BOTH the calibration and extrinsic seed paths. EXECUTE-POINT -- rebuild the container. demo/rift/calmarg: PP_CALPOST toggle (default 1) threads --calmarg-export-posterior into pp-run-build and extr-run-build, so the recovered cal posterior is written in the runnable demos. (Does NOT touch a running rundir_pp_run -- pp-run-build starts with its own rm -rf.) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…nders; works on old container) Complement to the ILE empty-breadcrumb size-guard: make the iteration-0 placeholder a VALID breadcrumb that LOADS cleanly, so the pipeline-writer fix ALONE (no container rebuild) keeps an older ILE binary from crashing on it. - generate_realizations.prior_cal_breadcrumb_dict(env_dir, dets, fmin, fmax, n_spline_points): build the 'cal' breadcrumb for the broad PRIOR with proposal == prior. Seeding from it draws cal realizations from the prior with ZERO importance weights -- exactly equivalent to the cold prior draws. Layout matches seed_realizations_from_breadcrumb (per-det [amp,phase] blocks; dim = 2N*len(dets)). - util_RIFT_pseudo_pipe.py: on OSG file-transfer, write cal_consolidated_-1.npz as that valid prior breadcrumb (was a 0-byte file) and extr_consolidated_-1.npz as a valid EMPTY breadcrumb (extrinsic=None -> cold). Falls back to a 0-byte file only if the build fails (then the ILE size-guard catches it). PIPELINE-WRITER change -- no container rebuild needed. - util_CalMakePriorBreadcrumb.py (NEW): (re)generate the prior placeholder for an ALREADY-built run dir IN PLACE (overwrite the 0-byte cal_consolidated_-1.npz), so an in-flight pilot run can be patched without re-running pseudo_pipe or rebuilding the container. Verified: the placeholder loads + seeds with max|cal_log_weights| ~ 1e-14 (== prior draws). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…lmarg flags The demo grew from the single-ILE correctness check into a ladder up to a runnable condor pipeline. Document all targets grouped by what they exercise (A: numerical correctness + single-ILE; B: direct-ILE DAG + tuning; C: offline pipeline build-validate incl. extrinsic handoff; D: runnable pipeline on CI data + pilots + extrinsic-handoff GPU run), the runnable toggles (OSG/PP_PILOT/PP_DMARG/PP_CALPOST/PP_NIT), the helper utils, and the advanced pipeline flags (--calmarg-export-posterior, --internal-marginalize-distance, --calmarg-pilot, --extrinsic-handoff). Add the recovered-cal-posterior section, the iteration-0 prior placeholder note (+ util_CalMakePriorBreadcrumb.py), and the execute-point vs pipeline-writer rule. Points to DESIGN_adaptive_driver.md / DESIGN_extrinsic_handoff.md. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Fit the LISA ecliptic sky as ordinary CIP coordinates (phi/theta) read from the NAMED hyperpipeline columns ecliptic_longitude/ecliptic_latitude, instead of the eccentric branch's positional all.net column-shift hack (which bakes in technical debt). Adds the reflected (secondary) sky mode as an in-loop DAG step. Both validated end-to-end on a CIT cluster smoke run (4-iteration ILE<->CIP loop, file-transfer container): vary-sky recovers masses+sky to the truth; the reflect node maps the primary mode to its secondary at iteration 2 and the next ILE consumes it cleanly. - util_ConstructIntrinsicPosterior_GenericCoordinates.py: ingest/fit/emit sky by named column (use_sky), prior_map/prior_range_map phi/theta, --phi-range/--theta-range. - helper_LISA_Events.py: vary-sky grid jitters sky off-lattice (breaks sky<->mass collinearity), CIP --parameter phi/theta + ranges, ILE per-row fixed sky. - convert_primary_sky_mode_to_secondary (new): hyperpipeline-aware sky reflection. - create_event_parameter_pipeline_BasicIteration / util_RIFT_pseudo_pipe.py: reflected-sky DAG node (gated before the target iteration's ILE) + forwarding. - LISA contract tests updated to the corrected design (named-column CIP params, per-point fixed sky, off-lattice sky spread); all 12 pass. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Allow util_RIFT_pseudo_pipe.py --use-ini to build the LISA known-sky surface, instead of hard-exiting. The generic [rift-pseudo-pipe] parser already maps every CLI arg by name (so all lisa-* scalars/algorithm options just work); a small _lisa_data_products_from_ini() fills the per-channel data products (channels, PSDs) from the conventional [data]/[lalinference] sections. No LDG data-find is invoked; the LISA branch stays self-contained. The ini path is a thin front-end over the validated CLI machinery: with matched config it renders a BYTE-IDENTICAL workflow (args_ile/args_cip_list/args_test and DAG, incl. the reflected-sky node). Verified against the CLI render that was itself validated end-to-end on a CIT cluster run. - bin/util_RIFT_pseudo_pipe.py: replace the --use-ini LISA hard-exit with _lisa_data_products_from_ini(); flat [rift-pseudo-pipe] lisa-* keys still win. - test/test_lisa_ini_contract.py: render ini vs CLI, assert byte-identical args + matching DAG/reflect node (locks the equivalence). - demo/rift/lisa/BBH_lisa_demo.ini: toy template (IMRPhenomD, vary+reflected sky). - demo/rift/lisa/run_lisa_ini_demo.sh: build inputs + instantiate + render. - demo/rift/lisa/README.md: document the production-ini path. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
… curvature
Adaptive re-whitening alone still left SNR640 collapsed: even seeded from the
tempered best sample, gradient-ascent MAP-polish stalled ~1615 nats below the peak
(on a 1/SNR-narrow peak the gradient is ~SNR^2-huge, so backtracking shrinks the
step to a crawl) -> the Fisher there was far too broad (theta-widths ~0.25 vs the
true ~1/640) -> still collapses. Add Newton refinement to _map_polish_4: after the
gradient phase, take curvature-normalized steps dx = F^{-1} g (F=-Hessian,
eig-floored pos-def) with backtracking line search; this lands on the sharp peak in
~1 step, so the subsequent Fisher whitening uses the TRUE peak curvature. n_newton
param (default 30). py_compile OK; testing SNR640 end-to-end.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…her-is-samples) The flow collapses at extreme SNR from NF-training nan, not because the posterior is unsampleable. Add a flow-free fallback: after annealing, draw N sky samples from the Fisher-whitened Gaussian about the (Newton-polished) MAP and importance-reweight by the true lnL (reuses _fisher_whitening + _gaussian_logq + evidence_from_logweights). Overrides exported samples; TI logZ stays primary (IS logZ reported as cross-check). CLI --fisher-is-samples (implies --fisher-precondition). py_compile OK. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…l) for high SNR Robust alternative to the flow that does not collapse on sharp peaks: a cloud of walkers climbs an adaptive temperature ladder; each rung tempered-resamples then applies K random-walk Metropolis 'puffball' moves whose proposal covariance is estimated FROM THE CLOUD (shrinks to match the posterior; never the Hessian, so slivers/non-Gaussianity can't fool it; re-broadens every rung -> cannot collapse). SMC analogue of RIFT-AV sample->puffball->sample / nested sampling. SMC evidence. CLI --smc-puffball/--smc-walkers/--smc-move-steps/--smc-puff-scale.
…n constant)
Now that SMC maps the posterior + finds the peak well, fit a moment-matched
('Fisher-like') Gaussian to the CONVERGED CLOUD (inflated for fat tails) and do one
importance-sampling pass -> a well-conditioned normalization constant
logZ = logmeanexp(lnL + log_prior - logq). Unlike the earlier Hessian/MAP Fisher-IS
(off-peak curvature -> ESS=1), this proposal matches the posterior the cloud already
mapped, so ESS is high. Forward-only (no AD). Reported as primary logZ; raw SMC logZ
kept in logZ_laplace. Params is_evidence/is_samples/is_inflate.
…en proposal fits) SNR640: cloud-IS gives ESS=18464/60000, logZ=204039.18 +/-0.006, agreeing with the raw SMC logZ to 0.1 nat and superseding the collapsed-flow per-sample TI (201962, biased ~2077 low). SNR40 (broad multimodal sky): single-Gaussian ESS=31 -> too crude, keep SMC logZ. Gate the IS-Z override on ESS>=max(500,2%N) so the high-SNR accurate evidence is used and the low-SNR poor fit is not.
…h drop)
Clearing sampler._rvs by setting each key to [] (rather than resetting the
dict) left a stale empty-list entry for any key added AFTER the fairdraw
subset -- notably 'integrand', set in mcsamplerAdaptiveVolume.integrate()
under --internal-use-lnL. In a batched run (n-events-to-analyze>1) the next
binary's integrate_log() fairdraw then did self._rvs['integrand'][indx_list]
and raised "list indices must be integers or slices, not ndarray", caught by
the per-binary handler and skipped. This silently dropped every binary after
the first in each ILE batch whenever fairdraw extrinsic output was enabled
(e.g. distance-slice export runs), decimating the grid ~99% (succ=1/job).
Resetting sampler._rvs = {} drops stale keys; integrate_log repopulates the
needed columns fresh each binary. Verified on a real GPU (cupy, AV sampler,
gwsignal SEOBNRv5PHM, l-max 4, --internal-use-lnL + --fairdraw-extrinsic-output
+ --resample-time-marginalization + --export-distance-slices): 3/3 batched
binaries now export, was 1/3.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
all-fresh slice centers come from quantile_slice_centers(), which uses fixed equi-probable quantiles (k+0.5)/K -- identical for every intrinsic, so K=1 pins every point to the median d and the union over the grid samples only the median locus. Add randomize=True: draw the K quantiles uniformly-in-CDF per call, so a single all-fresh slice per intrinsic becomes a fair-draw of d from THAT intrinsic's posterior. Over the intrinsic grid this samples (intrinsic,d) jointly -- cheap (~1 slice/intrinsic) dense coverage for a continuous AD surrogate -- while the dense fairdraw extrinsic posterior provides the reference. Verified numerically: K=1 randomized over 5000 draws reproduces the input d-posterior quantiles; deterministic K=1 stays on the median. Wired end-to-end: ILE --distance-slice-randomize -> pseudo_pipe --export-distance-slices-randomize -> cepp --last-iteration-export-distance-slices-randomize. Optional rng for repro. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…r_fairdraw_batch Rift o4d fix rvs clear fairdraw batch
The container-family manifest emits an expression-valued
MY.SingularityImage = ifThenElse(TARGET.GPUs_Capability ...). HTCondor's
native singularity (CIT-local startds) evaluates this against the slot ad,
but OSG/OSPool glidein pilots read SingularityImage as a LITERAL string and
hard-hold the job ("Unable to download or build singularity image
ifThenElse(...)"). Match-time $$()/ifThenElse also cannot see the real GPU on
opportunistic glideins.
Add an opt-in alternative that selects the container at JOB START on the real
device:
- container_manifest.build_runtime_selection_wrapper(manifest, inner_command):
generates a bash wrapper (cap->image table baked from the same manifest) to
be used as the Condor executable on the bare node (NO +SingularityImage). It
detects nvidia-smi compute_cap, picks the matching family image, acquires
ONLY that image (in-place CVMFS/local verbatim; osdf:// single stashcp/pelican
fetch; never the whole family), and execs the real in-container command via
nested apptainer.
- write_ILE_sub_simple: when RIFT_CONTAINER_RUNTIME_SELECT is set (and a family
manifest + use_singularity), emit the wrapper as the executable instead of
MY.SingularityImage / the $$() transfer token. The require_gpus capability
floor is still applied. Default (env unset) behavior is byte-identical: the
existing ifThenElse path is unchanged for CIT-local runs.
Tests: wrapper text/syntax, live capability->image selection (incl. fallback),
and integration asserting no MY.SingularityImage / no $$() token / floor
present / wrapper emitted with the right inner command. Existing CIT-local and
single-sif tests unchanged.
This is an investigation branch for review; CIP/PSD/calibration submit writers
still use the ifThenElse path (fine on CIT-local; would need the same wrapper
for a fully OSG run). CVMFS publication is intentionally not required (no direct
publication authority); the wrapper's default OSG acquisition is a single OSDF
fetch.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…bustness)
The DAG sizes per-iteration MARG chunks from n-samples-per-job, but the grid an
adaptive iteration actually places (e.g. a puffball) can be SMALLER, so the tail
chunks ran off the end and the marg driver raised SystemExit ("index range
[a,b) exceeds grid size N"). Clamp the range to the grid instead; a fully
out-of-range chunk yields an empty (header-only) output the consolidation
ignores. Surfaced running an adaptive Rapster (popsynth_hyperpipe) loop.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…0 = inf)
fit_rf used sample_weight=1/y_errors**2; a zero error (placeholder rows can leak
into the accumulated marg net with sigma=0) makes the weight infinite, which
sklearn rejects ("Input sample_weight contains infinity"). Floor sigma at 1e-3.
Surfaced in an adaptive Rapster (popsynth_hyperpipe) EOS-posterior fit.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…bustness)
The DAG sizes per-iteration MARG chunks from n-samples-per-job, but the grid an
adaptive iteration actually places (e.g. a puffball) can be SMALLER, so the tail
chunks ran off the end and the marg driver raised SystemExit ("index range
[a,b) exceeds grid size N"). Clamp the range to the grid instead; a fully
out-of-range chunk yields an empty (header-only) output the consolidation
ignores. Surfaced running an adaptive Rapster (popsynth_hyperpipe) loop.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…0 = inf)
fit_rf used sample_weight=1/y_errors**2; a zero error (placeholder rows can leak
into the accumulated marg net with sigma=0) makes the weight infinite, which
sklearn rejects ("Input sample_weight contains infinity"). Floor sigma at 1e-3.
Surfaced in an adaptive Rapster (popsynth_hyperpipe) EOS-posterior fit.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…rescue
When a dev run's rescue DAG gets zeroed (a resubmit bypassed it, a cluster hiccup,
an accidental fresh start), DAGMan restarts from scratch and redoes work whose outputs
are already on disk. The dagman.out is appended across every invocation and still logs
each finished node ("Node <name> job proc (...) completed successfully"), so this
reconstructs a partial rescue (v2.1.0) marking those DONE -- recursively for nested
SUBDAG EXTERNAL inner DAGs, and unioning in existing DONE marks to preserve manual fixes.
Validated recovering 519/2638 outer + 430/1948 inner nodes after a fresh-restart wiped
the rescue. Dev recovery tool; does not submit. Generalizes the by-hand mark-DONE trick.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
… from a run's posterior A cal pilot normally starts COLD (iteration-0 seed = prior placeholder, wide_0 draws cal realizations blindly). Any run with --calibration-export-posterior writes, at its final fairdraw, <out>_<event>_cal.dat: samples of the recovered cal posterior over the same spline-node vector the pilot breadcrumb is a Gaussian over. This fits those samples into proposal_mean/proposal_cov and writes them as cal_consolidated_-1.npz (reusing the placeholder's prior + node structure), so the pilot starts WARM -- seeded where the likelihood already constrained calibration. Reusable to harvest cal posteriors from ANY prior run (esp. high-SNR / well-constrained events, where prior-draw cal n_eff collapses). Validated end-to-end on synthetic samples (fitted proposal ~13x tighter than prior). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…t from production PE Like asimov bootstraps an intrinsic grid, a finished PE analysis already constrains an event's CALIBRATION -- so a cal pilot can seed from it instead of starting cold from the broad prior. Production pesummary/bilby posteriors carry the recovered cal posterior as recalib_<IFO>_amplitude_<k> (fractional dA/A) and recalib_<IFO>_phase_<k> (rad) -- the same spline-node vector a RIFT cal breadcrumb is a Gaussian over. This fits those samples into proposal_mean/proposal_cov and writes the pilot's iteration-0 seed cal_consolidated_-1.npz (reusing a placeholder's prior + node structure). Reads via pesummary.io.read (h5py fallback). Validated against REAL data (S240426s bilby SEOBNRv5PHM, 14392 samples, H1/L1 x 10 nodes): fitted proposal ~2.7x tighter than prior, centered on the recovered cal offset. COMPATIBILITY (enforced/warned): same detectors + spline count required (clear error on mismatch); spline node FREQUENCIES must match too (documented caveat -- seeds the same / cal-identical event's pilot, not a cross-event transfer). Complements calposterior_to_breadcrumb.py (which reads a RIFT in-loop _cal.dat). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Read-only, engine-agnostic query/pool layer over the Archive so inference reuses completed sims instead of regenerating: - iter_completed, param_distance (normalized, dict/scalar, ignores seed) - find_matching (same_q -> unbiased same-Lambda pooling / seed-averaging) - find_nearby (k-NN / tolerance, distance-annotated, approximate) - pool_catalogs (concat shape across sims+levels; mu reported per-level, never summed) + gather_samples reuse entry point Additive only (database.py untouched). 6 tests pass. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…worker Keep no-worker CIP grids in workflow root
…ikelihood Update JAX ILE likelihood work in calmarg
Update Ralph/LISA work in calmarg
The cal_env/H1.txt rule was a bare file-existence target, so it silently reused a stale cal_env after make_cal_envelopes.py's default envelope dropped from 5%/3deg to a realistic 1%/1deg. The stale 5-8% 'wide' envelope collapses cal n_eff to ~3 at SNR~17 by design, making the pp-run demo look permanently unconverged (sigma_lnL~0.6) when the code/estimator were correct. Add the generator as a prerequisite (force regen on default change) and echo the in-use amp/phase 1-sigma at build so a wide envelope is loud instead of silent. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
A single integrate_likelihood_extrinsic_batchmode invocation evaluates the
contiguous intrinsic-grid range [--event, --event+--n-events-to-analyze)
serially on ONE GPU. On clusters where whole multi-GPU nodes are reserved but
ILE only requests request_GPUs=1, the remaining GPUs sit idle (e.g. macrongroup
~100 points per job on a 4-GPU node uses 1/4 of the hardware).
ile_pre.sh now wraps the ILE executable in a small launcher that, when opted in,
splits that point range into N disjoint shards run concurrently -- one per GPU.
Each shard is pinned with CUDA_VISIBLE_DEVICES and given a distinct
--output-file prefix (<orig>.gpu<dev>), so the per-point output files
(<prefix>_<localidx>_.dat / .xml.gz) never collide. Downstream collection is
unaffected: util_ILEdagPostprocess.sh globs CME*.dat and util_CleanILE.py
de-duplicates by parameter value, not filename. The shards partition the range
exactly (sizes differ by <=1), so coverage is identical to the serial run; the
launcher's exit code is the first non-zero shard code, preserving condor
retry/hold behaviour (e.g. CUDA hard-fail 62).
Controlled by env var RIFT_ILE_GPU_FANOUT (propagated to jobs via getenv=*RIFT*):
unset / "0" / "1" -> no fan-out; the launcher exec()s the binary unchanged,
so default behaviour is byte-for-byte identical.
"auto" -> one shard per visible GPU (CUDA_VISIBLE_DEVICES, else
nvidia-smi); for a whole node held with request_GPUs=1.
<int N> -> up to N shards (capped by #GPUs and #points); the DAG
also requests request_GPUs=N and request_CPUs=N so
HTCondor assigns the devices.
Changes are mirrored in dag_utils.py and dag_utils_generic.py (each carries its
own copy of write_ILE_sub_simple). request_CPUs is threaded through the
singularity branch so the fan-out CPU count is not clobbered back to 1.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Make the fan-out usable through every front-end, including asimov (which builds
the DAG in a clean environment and cannot rely on RIFT_ILE_GPU_FANOUT being
exported in the submit shell):
- Bake the resolved fan-out value into the generated ile_pre.sh as
export RIFT_ILE_GPU_FANOUT="${RIFT_ILE_GPU_FANOUT:-N}", so the job needs NO
runtime environment (a runtime value still overrides). ile_invocation_shell()
now takes the value; ile_gpu_fanout_value() resolves it at build time. Mirrored
in dag_utils.py and dag_utils_generic.py.
- Add --ile-gpu-fanout to util_RIFT_pseudo_pipe.py and
create_event_parameter_pipeline_BasicIteration; both funnel it through
RIFT_ILE_GPU_FANOUT (pseudo_pipe runs BasicIteration via os.system, inheriting
the env), so request_GPUs/CPUs sizing and ile_pre.sh baking happen on one path.
Asimov needs no code change: a blueprint sets the value via
scheduler.environment variables: {RIFT_ILE_GPU_FANOUT: N} (rift.py copies it
into os.environ before running the pipeline) or
scheduler.pipeline: {ile-gpu-fanout: N} (-> CLI flag).
Demo: demo/rift/infra/multi_gpu/ (README, Makefile, CI ini, asimov blueprint +
frozen container-family pin, fake_ile stub).
- make smoke-local: builds a REAL ile_pre.sh from the shipped helper around a
stub ILE and runs it across this node's GPUs; asserts exact coverage, GPU
spread, distinct per-shard output prefixes. Runs anywhere (no cupy/condor).
- make build / make verify: builds a real pipeline run dir on the CI synthetic
data (singularity/OSG so ile_pre.sh is emitted) with --ile-gpu-fanout and
asserts ILE.sub gets request_GPUs=N/request_CPUs=N and ile_pre.sh bakes N.
Verified end-to-end: ILE.sub -> request_GPUs=4/request_CPUs=4, ile_pre.sh ->
RIFT_ILE_GPU_FANOUT:-4 wrapping the container ILE binary; default stays 1 (no-op).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add the two ways to request a VARIABLE number of GPUs (HTCondor's plain
request_GPUs is a single fixed count, so it cannot natively ask for "1 to N"):
- RIFT_ILE_GPU_FANOUT=auto-max-N (shared / partitionable-slot pools)
request_GPUs/request_CPUs become a ClassAd expression that asks for up to N of
the capability-matching GPUs available on the matched slot:
ifThenElse(countMatches(RequireGPUs,AvailableGPUs) >= N, N,
ifThenElse(... >= 1, ..., 1))
(same countMatches idiom RIFT already uses for cross-platform GPU matching).
ile_pre.sh bakes 'auto', so the launcher splits across exactly the 1..N GPUs
condor grants. Override the expression with RIFT_ILE_GPU_REQUEST_EXPR if your
pool exposes GPU counts under a different attribute.
- RIFT_ILE_GPU_FANOUT=all (dedicated / whole nodes you reserve)
keep request_GPUs=1 (matches a node with ANY GPU count) and have the launcher
enumerate EVERY physical GPU via nvidia-smi, ignoring CUDA_VISIBLE_DEVICES.
Launcher _devices(physical=True) drives this.
The runtime split was already adaptive (the launcher splits the point block across
however many GPUs it is handed); these add the matching request side. Implementation:
ile_gpu_fanout_count() -> ile_gpu_request() returning (request_gpus, request_cpus) as
an int OR a ClassAd expression; ile_gpu_fanout_value() maps auto-max-N -> baked 'auto'.
Mirrored in dag_utils.py and dag_utils_generic.py.
Verified: launcher splits across 1/2/3/4 granted GPUs (full coverage each); 'all'
uses all 4 physical even with CUDA_VISIBLE_DEVICES=0; generated ILE.sub carries the
adaptive expression for auto-max-4, request 1 for 'all', fixed N for N.
Demo: new `make requests` shows request_GPUs/CPUs + baked launcher for each mode;
README "Values -- fixed vs. adaptive" documents the hot-swap options, the
partitionable-slot requirement, and the cgroup/reservation caveats; blueprint shows
the adaptive variants.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…back Per deployment policy (whole nodes are reserved; partitionable GPU slots are not a sustainable long-term path), make the GPU multi-GPU policy default to 'all': - DEFAULT_ILE_GPU_FANOUT = 'all'. When RIFT_ILE_GPU_FANOUT is unset, a GPU ILE job keeps request_GPUs=1 (matching unchanged -- lands on any GPU node exactly as before) but the launcher enumerates EVERY physical GPU (nvidia-smi, ignoring CUDA_VISIBLE_DEVICES) and splits the ILE block across all of them. On a 1-GPU node this is a no-op; only multi-GPU nodes change. - Fallback to the old single-GPU run: RIFT_ILE_GPU_FANOUT=1 (or 'single'/'off', or --ile-gpu-fanout 1). Aliases handled in the resolver and the launcher. - Safety: the bake is gated on request_gpu, so a CPU-only ILE job always bakes '1' and never grabs the node's GPUs under the 'all' default. Implementation: shared _raw_ile_gpu_fanout() applies the default + single/off aliases; ile_gpu_fanout_value()/ile_gpu_request() build on it; write_ILE_sub_simple passes fanout=(ile_gpu_fanout_value() if request_gpu else '1'). Mirrored in dag_utils.py and dag_utils_generic.py. Note: HTCondor partitionable GPU slots DO work today (verified on the CIT pool: a 2-GPU partitionable slot carves per-GPU dynamic slots), so auto-max-N remains available, but it is no longer the recommended/default path. Demo updated: README "Default policy" + Values table (all=default, 1/single=fallback); `make requests` shows default vs fallback; blueprint defaults to no override. Verified: default bakes 'all' and runs across all 4 physical GPUs even with CVD=0; RIFT_ILE_GPU_FANOUT=1 runs single; CPU-only job bakes '1'; fixed N and auto-max-N unchanged. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…_fanout Rift o4d multigpu ile fanout
Owner
Author
|
Considerable dev testing at this point, so likely not breaking critical-path outcomes. This is into the main dev branch, so perfect safety is not guaranteed |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
calmarg done in the ILE loop, including
as well as fancy tools to