Add dependency compatibility CI checks#155
Open
oshaughnessy-junior wants to merge 46 commits into
Open
Conversation
…e ini) Provide an end-user ini shape that carries BOTH multi-GPU fan-out AND the multi-container "container family" (per-machine .sif selection by GPU capability), and confirm every option lands in the generated ILE.sub. pseudo_pipe changes so a single ini is self-contained: - Let the ini [rift-pseudo-pipe] section set the container family via singularity_rift_image / singularity_base_exe_dir -> SINGULARITY_RIFT_IMAGE / SINGULARITY_BASE_EXE_DIR (raw strings into os.environ, env still dominates), mirroring the existing ile_require_gpus -> RIFT_REQUIRE_GPUS hook. Read from the environment by create_event_parameter_pipeline_BasicIteration / write_ILE_sub_simple. - Move the --ile-gpu-fanout -> RIFT_ILE_GPU_FANOUT funnel to AFTER ini parsing, so an ini-only ile-gpu-fanout=... value also propagates (previously only a CLI value did). Demo (demo/rift/infra/multi_gpu/): - multigpu_multicontainer.ini: annotated template with fan-out (ile-gpu-fanout="all", ile-force-gpu, ile-jobs-per-worker) + container family (use-osg/use-singularity, singularity_rift_image, singularity_base_exe_dir, ile_require_gpus). Documents the ini eval() quoting rule (string values quoted; raw-string env keys not). Silly IMRPhenomD zero-spin CI config so it can build/run. - make build-combo / verify-combo: build from the ini with NO container/fan-out env vars set, then assert BOTH landed -- MY.SingularityImage ifThenElse per-machine selection, selective osdf .sif transfer, combined require_gpus floor, request_GPUs/CPUs, and ile_pre.sh baking RIFT_ILE_GPU_FANOUT. Verified end-to-end: pseudo_pipe parses the ini (ile_gpu_fanout='all', use_osg/ use_singularity True, USING SINGULARITY); the generated ILE.sub carries the per-machine container selection + capability floor + osdf transfer AND the fan-out; verify-combo passes. (The build's grid step can hit a pre-existing flaky scipy import; documented -- just re-run.) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Realizes the response coefficients for the slow-rotation RIFT generalization,
derived from the exact conventions in vectorized_lal_tools.py:
F(t) = F_+ + i F_x = sum_{n=-2}^{2} A_n exp(i n g) (5 harmonics)
tau(t) = -(1/c) r.nhat = sum_{n=-1}^{1} B_n exp(i n g) (3 harmonics)
with g = GMST(t) - RA. The complex antenna harmonics A_n depend only on the
detector response tensor, declination, and polarization (NOT RA or time); the
delay harmonics B_n depend only on detector location and declination. RA and
time enter solely through exp(i n g) = exp(i n (GMST(t)-RA)), i.e. the sidereal
modulation exp(i n Omega t) of the design notes, keeping sky in analytic scalars.
test_slowrot_response.py validates the reconstruction against
lal.ComputeDetAMResponse and lal.TimeDelayFromEarthCenter over a full sidereal
day for H1/L1/V1/K1 and random sky/polarization: agreement to machine precision
(|dF| ~ 4e-16, |dtau| ~ 9e-18 s), and confirms no harmonic content beyond the
predicted bands.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
New realization of factored_likelihood.PrecomputeLikelihoodTerms that folds the
time-varying detector response into the likelihood without regenerating templates.
Each FD mode is built ONCE; the two response modulations are cheap FD operations:
* delay-derivative order p: h^(p)(t) <-> (FT_SIGN 2 pi i f)^p H(f) (exact per-bin weight)
* sidereal harmonic n: exp(i n Omega t) <-> sub-bin frequency shift, realized as an
exact LAL-FFT time-domain phase (no integer bin roll)
PrecomputeLikelihoodTermsWithRotation returns the intrinsic-only, sky-independent overlap
bank Q^a_lm(t), U^{a,a'}, V^{a,a'} keyed by elementary templates a=(p,n), reusing the
existing ComputeModeIPTimeSeries / ComputeModeCrossTermIP unchanged. For the data term it
uses the mode-independent optimization of shifting the DATA by -n f_sidereal (one shift per
(det,n)) rather than modulating each template. Response coefficients A_n,B_n that contract
these come from slowrot_response (computed in the extrinsic layer, not here). Path A
(p_max=0) is the default; Path B is p_max>=1. Heavy RIFT imports are lazy so the FD
primitives are importable with numpy alone.
test_slowrot_fd_ops.py validates the primitives against LAL FFT round trips (numpy+lal
only): round-trip identity 9e-16; FT_SIGN=-1 pinned down from the tone-assignment convention
(evaluate_fvals reverses omega sign); time derivatives p=1,2,3 exact to ~1e-13; sidereal
modulation exact to 5e-16; O(N^2) reference agrees with the LAL round trip to 4e-15.
Full-stack V1 validation (agreement with a brute-force time-varying-response likelihood)
needs the data/waveform environment and is the next step.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Full-stack integration testing (test_slowrot_precompute_integration.py, run against the
ILE-GPU-Paper-style noiseless-injection data + analytic PSD, same machinery as the CI demo)
exposed a reference-time bug: the sidereal modulation used local array time (from 0,
ignoring the series epoch) plus a spurious arrival-time post-phase, leaving an O(Omega*seglen)
~ 1e-4 error.
Fix: apply the modulation at ABSOLUTE data time referenced to the event time,
exp(-i n Omega (t' - t_ev)) on the data (data route), with NO post-phase. This is the
physically correct factorization of the true antenna phase
exp(i n (GMST(t')-RA)) = exp(i n (GMST(t_ev)-RA)) * exp(i n Omega (t'-t_ev)),
the first factor carried analytically by A_n (slowrot_response), the second by the
precompute. _lal_freq_modulate now takes t_ref and uses the series epoch; U,V templates use
the same t_ev reference. Dropped the modulate_data flag (the data route is the single
correct path).
Validation on real waveforms/data:
V0 (recovery): rotation precompute with harmonics=(0,), p_max=0 reproduces the baseline
PrecomputeLikelihoodTerms exactly: rel|dQ|=0, |dU|=0, |dV|=5e-13.
Vmag: with harmonics -2..2, a=(0,0) still matches baseline exactly; modulated a=(0,+-1)
and a=(0,+-2) differ from n=0 by 7.3e-6 and 1.47e-5 -- nonzero, at the Omega*duration
scale, and exactly linear in n (n=2 is 2x n=1) as the modulation demands.
FD primitive unit test still passes (t_ref defaults preserve it).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add FactoredLogLikelihoodWithRotation and antenna_harmonics_tilde: Path A
(amplitude-drift) log-likelihood contracting the harmonic precompute bank with
A_tilde_n = A_n exp(i n (GMST(tref)-RA)) from slowrot_response. The V-term's
A_tilde_{-nu} (not A_tilde_nu) matches the precompute V^{(nu,n')} exponent (n'-nu) to
the F^2 exponent (n+n'); both terms reduce exactly to baseline as f_sidereal -> 0.
V1 (test_slowrot_likelihood_v1.py) on the ILE-GPU-Paper-style injection:
V1a: f_sidereal=0 rotation lnL == baseline FactoredLogLikelihood to 2.7e-12.
V1b: real sidereal rate rotation lnL == independent brute-force time-varying-response
likelihood to 1.6e-9 (rel ~3e-13).
Path B (delay-derivative contraction with B_n) remains a TODO.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Wire Path A into the maintained vectorized likelihood path.
* slowrot_response.antenna_harmonics_vector: vectorized antenna harmonics A_n over
arrays of (dec, psi) (uses np.broadcast_shapes for scalar/0-d/array uniformity).
* factored_likelihood_with_rotation.pack_rotation_arrays: pack the harmonic-indexed
bank Q^(n), U^(n,n'), V^(nu,n') into dense (n_lms,*) arrays (per-harmonic analogue of
PackLikelihoodDataStructuresAsArrays).
* factored_likelihood_with_rotation.DiscreteFactoredLogLikelihoodViaArrayVectorNoLoopWithRotation:
the vectorized rotation lnL, mirroring DiscreteFactoredLogLikelihoodViaArrayVectorNoLoopOrig
with sums over sidereal harmonics and per-sample A_tilde_n = A_n exp(i n (GMST(tref)-RA)).
* factored_likelihood.py: fix the non-numba fallback for lalylm. numba's @vectorize
cannot JIT the LAL swig call, so RIFT falls back to a plain scalar lalylm that breaks
ComputeYlmsArrayVector (it passes numpy ARRAYS) for BOTH the baseline and rotation
vectorized paths. np.vectorize replicates the numba elementwise behavior. Needed for
the vectorized ILE to run at all in non-numba / non-GPU environments.
Validation (test_slowrot_noloop.py, test_slowrot_noloop_bruteforce.py; real
noiseless-injection data):
(A) antenna vectorized-vs-scalar: 1.2e-16.
(B) rotation NoLoop == baseline NoLoop at f_sidereal=0: max|dlnL_t| = 3.6e-12.
brute-force: rotation NoLoop lnL_t == an independent true-time-varying-response
likelihood (F_k(t) from lal.ComputeDetAMResponse, applied to data and modes) over the
whole window: max|d| = 5.9e-9. This also confirms the large marginalized-lnL shift
the sidereal rate induces at high SNR is genuine physics, not a bug.
Still TODO: wire --rotation-slow into integrate_likelihood_extrinsic_batchmode.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…ion-slow into batchmode ILE CRITICAL FIX (corrects a bug that the prior commit's brute-force check shared and so missed): the U,V cross-term templates chi were modulated with t_ref = event_time (~1e9) against the template's INTRINSIC epoch (~ -T_dur, near 0), applying a spurious exp(i n Omega * ~-1e9) ~ 1e4 rad phase that randomized U,V and inflated lnL beyond the Cauchy-Schwarz bound 0.5<d|d>. The data-term route was fine (data epoch ~ event_time). Fix: reference the template modulation to 0, so it is exp(i n Omega (hlms.epoch + j*dt)) = exp(i n Omega (t' - t_ev)), the correct small intrinsic-time modulation. Effect: the genuine slow-rotation shift for a short 30+25 BBH collapses from a spurious ~660 in lnL to ~0.008 (rel 3e-6) -- negligible, as physics requires. The earlier "660 is real physics" conclusion was WRONG; it was entirely this bug (the brute-force reference sampled template F at the same wrong absolute time, hence agreed with the buggy code). Brute-force tests corrected to sample template F at event_time + hlms.epoch. Re-validated: V1a (Omega=0 vs baseline): 2.7e-12; V1b (real Omega vs fixed brute force): 2.6e-9, and rotation lnL now ~ baseline (tiny shift); NoLoop B (vs baseline at f_sid=0): 3.6e-12; NoLoop C (f_sid=0 vs real): diff 0.008 (was 660); NoLoop brute-force: 3.2e-10. Wiring: --rotation-slow (+ --rotation-n-harmonics) added to integrate_likelihood_extrinsic_batchmode, guarded (requires --vectorized, CPU, no dist/phase marg). After the baseline precompute+pack it builds+packs the rotation bank; the CPU vectorized likelihood_function calls DiscreteFactoredLogLikelihoodViaArrayVectorNoLoopWithRotation. End-to-end on the ILE-GPU-Paper demo data: baseline unaffected; --rotation-slow runs cleanly (no inf/nan) and gives baseline-consistent marginalized lnL for the short signal. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…ire --rotation-p-max
Path B accounts for the slow drift of the propagation delay tau_k(t) over the signal, via
a time-domain derivative expansion. The Path A antenna coefficient A_tilde_n generalizes to
C_{(p,ntilde)} = (1/p!) sum_{n+m=ntilde} A_tilde_n [(-D)^{*p}]_m
where D_m are the delay-DRIFT harmonics (delta_tau = tau(t)-tau(tref), D_0=-(B_tilde_1+
B_tilde_-1), D_{+-1}=B_tilde_{+-1}). The likelihood is the Path A structure with
A_tilde_n -> C_{(p,n)} summed over all elementary templates a=(p,n), V-term reflecting the
harmonic (C_{(p,-nu)}). At Omega->0, delta_tau=0 so the p>=1 coefficient sums vanish and
Path B -> Path A -> baseline.
* slowrot_response.delay_harmonics_vector; factored_likelihood_with_rotation:
_convolve_harmonics, rotation_coefficients (scalar) + rotation_coefficients_vector.
* FactoredLogLikelihoodWithRotation and DiscreteFactoredLogLikelihoodViaArrayVectorNoLoop-
WithRotation generalized from n-keyed to a=(p,n)-keyed (pack_rotation_arrays too).
* ILE: --rotation-p-max (widens the precompute harmonic range to cover all C_{(p,ntilde)}).
Validation (test_slowrot_pathB.py; Path A tests still pass):
scalar Path B (p_max=1, f_sid=0) == baseline FactoredLogLikelihood: 9.1e-13;
scalar Path B (p_max=1, real) respects Cauchy-Schwarz 0.5<d|d> at injection;
vec Path B (p_max=1, f_sid=0) == baseline NoLoop: 6.4e-12.
End-to-end: --rotation-slow --rotation-p-max 1 runs cleanly (no inf/nan), baseline-consistent
for the short signal. Rigorous p>=1 validation vs a time-varying-DELAY brute force on a LONG
signal is deferred to the systematic final-validation pass.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
test_slowrot_headtohead.py draws ONE fixed set of extrinsic samples (fixed RNG seed,
standard cos/ d^2 priors) and evaluates BOTH the baseline vectorized (NoLoop) likelihood
and the rotation (NoLoop) likelihood on the SAME samples + SAME time grid, so the
difference is the genuine slow-rotation effect with zero Monte-Carlo noise.
Checks (30+25 BBH, SNR~130, short signal):
(R) regression: rotation at f_sidereal=0 == baseline to 9.1e-13 on the finite-baseline
samples (the baseline NoLoop uses a GLOBAL lnLmax offset, so uniform-prior samples
far below the global peak underflow to -inf; our rotation NoLoop uses a per-sample
offset and is always finite).
(P) head-to-head: rotation(real) vs rotation(0), matched samples: per-sample max |dlnL|
= 0.052; evidence shift ln Z_rot - ln Z_base = -1.1e-3 (MC-noise-free) -- negligible
for a short signal, as physics requires.
(B) Cauchy-Schwarz: max lnL 3707 < 0.5<d|d> = 8421.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…delay-physics still to validate
Inflated-Omega convergence (single-det 30+25 BBH, f_sid x3000): scalar Path B
lnL(p_max=0..3)=[1794.39,1781.31,1781.63,928.30] -- p=0..2 capture a real ~13-in-lnL delay
effect and appear to converge (~1781.6) within 0.5<d|d>=1938, but p_max=3 blows up (853).
Root cause: FD derivative (2 pi i f)^p amplifies high f; model-norm integrand ~ f^{11/3} for
a chirp, dominated by the f_max edge. Fix: band-limit the p>=1 delay-derivative terms to low
frequency. Independent time-varying-delay brute force still required (convergence alone can
converge to a wrong value -- the delay analogue of the reference-time bug).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…ath B delay validation Findings: RIFT data already uses SimInspiralTD + SimDetectorStrainREAL8TimeSeries (full Earth-rotation delay map). The correct rotation ground truth applies SimDetectorStrain to the SAME RIFT modes via lsu.hoft_from_hlm(hlmsT,P) + ComplexIP -- NOT non_herm_hoff (which regenerates via SimInspiralTD; its modes differ from hlmoff, so with IMRPhenomD the mode-baseline recovers only ~SNR 35 of the data's ~82 at injection: a waveform-convention mismatch, not rotation). Gotchas: pad/align hoft_from_hlm output to the data length; compare time-maximized (fixed-tref is off-peak). Use a long signal (real rate) so delay drift matters. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…) + SimDetectorStrain check test_slowrot_pathB_bruteforce.py: independent folded-template ground truth Re[F(t) Sigma(t-tau(t))] from the same harmonic F(t),tau(t) + same modes as the likelihood (no SimDetectorStrain floor), inflated sidereal rate. At the real 90-min-BNS rate (x340 on a 16s test) Path B CONVERGES: p_max=0 deficit 3.43 -> p=1 0.23 -> p=2/3 0.207, bound-respected, no blow-up. Delay drift recovered; production-ready for target signals with p_max<=2. Catastrophic p>=3 cancellation only at x1000+ (>2.6x any physical signal) -> band-limit fix is a robustness nicety, not a blocker. test_slowrot_pathB_groundtruth.py: vs LAL SimDetectorStrainREAL8TimeSeries (Jolien's full delay map), built from the likelihood's OWN modes. baseline/PathA/PathB agree with LAL to ~0.07 at fmax=256; the ~26 deficit at fmax=1024 was SimDetectorStrain's high-f TD delay-interpolation (confirmed: vanishes as fmax lowers), not a Path A/B bug. HANDOFF updated with the validation summary and the known p>=3 limit. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…ependent response elevated to active)
… + validation
slowrot_freqresponse.py: closed-form ground-based antenna response beyond the long-wavelength
approximation, F_k(f; RA, DEC, psi), from arXiv:2412.01693 (Rakhmanov-Romano-Whelan) single-arm
round-trip transfer D~(a,n,f) = (e^{-i2pi fT}/2)[e^{+i pi fT_-} sinc(pi fT_+) + e^{-i pi fT_+}
sinc(pi fT_-)], T=L/c, T_pm=T(1 +- a.n). Written as the EXACT LAL long-wavelength response plus
a finite-size correction that is identically zero at f=0 (LAL stores d.response in float32, so
reconstructing arms directly would cap f=0 agreement at ~1e-7).
test_slowrot_freqresponse.py (numpy+lal, loadable by path):
(A) f->0 == lal.ComputeDetAMResponse to 6.66e-16 over 32 random (ra,dec,psi,gmst) x {H1,L1,V1,K1};
Im F(0)=0 exactly.
(B) free-spectral-range: for a.n=0, D~=e^{-i2pi fT} sinc(2fT) exactly, first null at f_FSR=c/2L
=3747.4 Hz (40 km); general a.n -> dips at 0.8-1.0 f_FSR.
(C) in-band pattern-SHAPE distortion ||F(f)|-|F(0)||/|F(0)|: LIGO 4km 0.24%@1KHz 0.62%@2khz;
CE 40km 11.8%@1KHz 42%@2khz (CE/LIGO ~68x ~ (10x arm)^2). => negligible for LIGO,
first-order only for 3G/CE. The complex ratio is dominated by a benign common
e^{-i2pi fL/c} delay (degenerate with coalescence time), NOT sky-dependent shape.
LAL has NO closed-form FD response (only ComputeDetAMResponse = LWL), so validated vs the LWL
limit + analytic FSR structure. Recommended likelihood route: (b) sky-harmonic expansion for CE
(keeps sky extrinsic; fold the common delay into geocenter time), (a) pinned-sky TD fold reserved
for precessing+HM. Uncommitted -> now committed after independent test verification.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…erify-anywhere demo done
…nic) + validation
factored_likelihood_freqresponse.py: finite-size analogue of factored_likelihood_with_rotation.
Factor out the benign common e^{-i2pi fL/c} delay (fold into arrival time); power-series the
residual single-arm transfer in the finite-size parameter -> sky-independent basis W_p(f) folded
into the FD modes (reusing ComputeModeIPTimeSeries/ComputeModeCrossTermIP) x analytic sky/pol
scalars b_p(RA,DEC,psi) -> SKY STAYS EXTRINSIC. Provides the MAINTAINED-STYLE NoLoop entry point
DiscreteFactoredLogLikelihoodFreqResponseNoLoop (+ a scalar companion FactoredLogLikelihoodFreqResponse).
No SingleDetectorLogLikelihood; validates against the maintained NoLoop.
Validation (test_slowrot_freqresponse_likelihood.py; CE 40 km, 16 s, verified by re-running):
V1 reduce-to-baseline: finite-size NoLoop (L->0) == DiscreteFactoredLogLikelihoodViaArrayVectorNoLoop
to 3.3e-9.
V2 ground truth: on an exact finite-size CE injection, the long-wavelength NoLoop leaves deficit
2.71; the finite-size NoLoop recovers it to 0.558, CONVERGED by Qmax=2 (basis 4). Residual
~= the NoLoop nearest-neighbor peak-resolution floor (cf. time-interpolated NoLoop in
oshaughn/rift_O4d).
V3 Cauchy-Schwarz: all lnL <= 0.5<d|d> (worst excess -0.558).
Scalar vs NoLoop freq-response agree to 0.156 at the peak (interp-vs-nearest floor).
NOT yet done: ILE wiring (rotation only so far), full precessing+HM (pinned-sky route a).
Docstrings corrected to name the maintained NoLoop, not the old scalar path.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…all slowrot likelihoods route via NoLoop, no SingleDetectorLogLikelihood)
…-preserving time ref
…bin time-ref precision (match baseline); tests to consistent cubic
…e +38.9 nats); fix stale docstring numbers
…tation-slow; thread --interpolate-time into slow-response NoLoops; freqresponse test V4 positive control
…) and finite-size (Path D) likelihoods
…c validation Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
… A/B rotation + Path D finite-size)
Squashed feature import onto rift_O4d (dev history on rift_slowrot / calmarg_in_loop).
Adds RIFT/likelihood/{factored_likelihood_with_rotation,factored_likelihood_freqresponse,
slowrot_response,slowrot_freqresponse}.py, test_slowrot_*.py, and demo/rift/slowrot/.
Wires --rotation-slow/--rotation-p-max (Path A/B) and --freqresponse (Path D) into
integrate_likelihood_extrinsic_batchmode (CPU vectorized/NoLoop, guarded, opt-in).
Validated: rotation NoLoop==baseline 3.6e-12; finite-size L->0==baseline 3e-9, Cauchy-Schwarz
respected, V4 positive control +38.9 nats; cubic time-interp with precision-preserving time ref.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…I/version on O4d, +multi_gpu demo on calmarg) Both branches now carry the same content; refs point to this commit so parallel agent teams see one canonical state. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…rinsic zoom-box options (--limit-right-ascension/-declination/-inclination/-psi) Enables the finite-size 3G sky-localization study: per-detector arm lengths for mixed CE+ET networks (LAL's cached C1 arm is a placeholder), and truth-centered sampling/prior restriction so the AdaptiveVolume sampler can resolve narrow high-SNR extrinsic peaks. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…lihood term1 reuses the baseline fused Q_inner_product cubic/nearest kernel PER elementary template (A=conj(Ylm)) so there is NO (n_ext,npts,n_lms) temporary -- same GPU memory footprint as the baseline, just looped over |a_list| (~5 for Path A, ~15 for Path B); term2 is small |a_list|^2 einsums over n_lms^2. Antenna-harmonic / Ylm / time-delay stay on host (numpy/LAL) and only the heavy Q banks + U/V move to device. ILE: --rotation-slow now allowed with --gpu (requires n_cal=1, no glitch/cal marginalization); precompute moves rho/U/V to cupy; GPU likelihood_function gains the rotation branch. CPU path bit-identical (test_slowrot_noloop still 3.6e-12). GPU<->CPU consistency: test_slowrot_gpu.py (skips without cupy; run on a GPU node). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…py rotation likelihood Runs test_slowrot_gpu.py (xpy=cupy vs xpy=np, <1e-8) on a real GPU inside the RIFT container with the branch code transferred (no cupy/CUDA in the dev sandbox). make submit / make show; env-driven (SINGULARITY_RIFT_IMAGE, RIFT_REQUIRE_GPUS, LIGO_ACCOUNTING/USER_NAME). See BREADCRUMB.md. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…pe, unstick matching) The 16384 default was sized for the wide-envelope stress case (sampler spinning to --n-max, ~7.3 GB completers). On the DEFAULT mild 1% envelope, measured ILE RSS is ~1.2 GB, so 16384 was ~13x over-provisioned -- and it mattered: the RAM-starved GPU pool has few >=16 GB slots, so 16384 (with a cc>=8 require_gpus floor) let jobs match only ~55 of 3400 slots and they idled for weeks. 6144 keeps ~5x headroom, matches the whole GPU pool, and (paired with a require_gpus floor of 6.0) admits the ~529 uncontested cc6.1 slots. --wide stress runs should bump back to 24576. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
… freqresponse to GPU
Validated on a real A100 (ldas-pcdev12, cupy 13.6/CUDA 11.8) via the container kit:
Path A/B rotation and Path D freqresponse GPU<->CPU parity 7e-12 (< 1e-8), CPU
paths bit-identical.
- factored_likelihood_with_rotation / _freqresponse: pin xpy=np on the host-side
TimeDelayFromEarthCenter call. It defaults xpy=xpy_default (=cupy whenever cupy is
importable), so passing host RA/DEC arrays crashed cupy.cos on a real GPU -- invisible
in a no-cupy sandbox, fatal on hardware.
- factored_likelihood_freqresponse: full GPU port mirroring the rotation NoLoop --
xpy=np arg, fused Q_inner_product_{,cubic}_cupy term1 per basis weight p, device
einsums for term2, optimized_gpu_tools.simps. CPU path unchanged (V4 +38.88 nats).
- integrate_likelihood_extrinsic_batchmode: --freqresponse now accepts --gpu (n_cal=1,
no glitch/cal marg); precompute moves Q/U/V to device, GPU likelihood_function gains
the freqresponse branch; help text updated on --rotation-slow/--freqresponse.
- test_slowrot_freqresponse_gpu.py: new GPU<->CPU parity test (mirrors test_slowrot_gpu).
- demo/rift/slowrot_gpu_validate: run_gpu_test.sh + Makefile now cover both paths;
new `make local-gpu` target runs directly in the container on a local GPU (no Condor);
BREADCRUMB records the validation + the SINGULARITY_RIFT_IMAGE env gap.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…se host-copy of extrinsic params Found running the real ILE (--rotation-slow/--freqresponse with --gpu) end-to-end in the cupy container -- three bugs invisible to a CPU-only sandbox, all now fixed. End-to-end CE-ET SNR30 (n_eff~130k): GPU vs CPU marginalized lnL agree to <0.5 sampler sigma for both paths (freqresponse dlnL=0.005, rotation dlnL=0.085; sigma~0.17), GPU 2-4x faster. - mcsamplerAdaptiveVolume.prior_prod: pass xpy=numpy to prior_pdf helpers that accept it. Several mcsamplerGPU prior helpers default xpy=cupy, so evaluating them on the host CPU copy (as prior_prod does by design) fed a numpy array to cupy.sin/ones and raised "Unsupported type numpy.ndarray" -- broke ANY AV-sampler run in a cupy-enabled container. - mcsamplerGPU: cupy_pi = np.pi (was cupy.array(pi)). A device scalar in the host-evaluated prior helpers made numpy_array/cupy_pi re-dispatch to cupy and raise; a plain float works for both backends (cupy_array/float stays on device). - factored_likelihood_freqresponse: extract P_vec extrinsic params via a _h() host-copy (mirrors the rotation NoLoop). Under --gpu the ILE hands RA/DEC/incl/... as cupy arrays, and np.atleast_1d(cupy) raised in ComputeYlmsArrayVector. The unit test never caught this (its P_vec is numpy); only the real ILE GPU path exposed it. CPU paths bit-identical; all slowrot GPU/CPU unit tests still pass. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
… + the 3 sampler/host-copy bugs Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…t + document 2 invocation gotchas
- run_e2e_consistency.sh + e2e_mkargs.py: run the real ILE {rotation,finite}x{cpu,gpu} on an ILE case
(default the finite-size CE-ET SNR30 inputs; RIFT_E2E_CASE override) and assert CPU-vs-GPU marginalized
lnL agree within TOL_SIGMA (4) x sampler error. Verified PASS on the A100 (rotation |d|=0.12, freqresponse
|d|=0.13, both <4 sigma). Makefile: `make e2e` runs it in the container on this host's GPU.
- Document the two LONGSTANDING, RIFT-wide invocation gotchas (aware, left as-is) in demo/rift/slowrot/README.md
(new "Running on GPU" section) and the BREADCRUMB:
1. --force-xpy alone is INERT -- opts.gpu stays False unless --gpu is also passed (and cupy absent);
pass --gpu to actually run the GPU code path.
2. the pure-CPU path (no --gpu) fails in a cupy container with "Unsupported dtype float128"
(legacy CPU-vectorized RiftFloat=float128); run --gpu inside a cupy container.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…n injection) make_e2e_inputs.py generates a throwaway H1/L1 IMRPhenomD BNS injection (frames via lsu.hoft + analytic PSD xmls + a 1-point util_ManualOverlapGrid grid + case.json with a 40-km arm) so the end-to-end CPU-vs-GPU consistency check has NO paper-repo dependency. run_e2e_consistency.sh now auto-generates this by default; set RIFT_E2E_CASE=/path to reuse an existing ILE case dir instead. Verified PASS on the A100 (self-contained): rotation dlnL=0.039, freqresponse dlnL=0.005, both < 4 sigma. Docs (BREADCRUMB, demo/rift/slowrot/README.md) updated to say self-contained. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…7 files) The validation-kit commit (bc53a86) accidentally tracked numba's on-disk JIT cache (SWSH_*/WignerD_*/qnm_*/... hash-named dirs full of *.nbc/*.nbi) written when tests ran in this dir with NUMBA_CACHE_DIR=. These are regenerable build artifacts -- the kit sets a fresh NUMBA_CACHE_DIR per run and does not depend on them. Untrack them and extend .gitignore so they never return. No functional change; keeps the PR diff to the real code/docs instead of ~2 MB of binary cache. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
A sample whose likelihood is -inf across ALL cal draws gives lnL_marg=-inf, so log_r = lc - lnL_marg = (-inf) - (-inf) = NaN, which poisons log_a and hence a_c / sigma_cal / neff_cal (and the adaptive-doubling stop test) for the ENTIRE batch. Such samples carry zero posterior weight, so force their responsibilities to -inf (contribute nothing) rather than NaN; a fully-degenerate batch returns a finite (inf, 1.0, uniform) sentinel instead of NaN. A pathological extrinsic point (near-zero antenna response) can trigger this on a real run. Adds test_total_underflow_guard; all existing tests + the adaptive self-test still pass unchanged. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…portable across numpy) Saving cal_dets and ext_g*_params as dtype=object pickled them into the .npz, making the breadcrumb numpy-version-fragile: a seed written by the container's numpy 2.x (cal_consolidated_N.npz from CALPILOT) fails to load on the host's numpy 1.14.3 with 'ModuleNotFoundError: numpy._core' (and the warm-start host tools' breadcrumbs could fail the other direction). The pipeline works container-side, but host tooling/inspection and any host<->container hand-off break. Store them as string (<U) arrays instead -- no pickle, loads under any numpy. Adds an allow_pickle=False assertion to the round-trip self-test so the regression can't return. All calmarg self-tests (breadcrumbs/pilot/adaptive) pass. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Walkthroughs to reconstruct the whitened detector strain with a 90% credible
band from RIFT ILE extrinsic fair-draw samples (cf. Fig 2 of arXiv:2009.05461).
- nr/ : fixed NR simulation vs an event (GW190521, RIT-Five eBBH-1794)
- model/ : self-contained full RIFT run (GW150914, IMRPhenomD aligned/zero-spin,
data from GWOSC) -> posterior -> reconstruction
- shared : reconstruct_strain.py (NR or --approx model back-end; each fair-draw
sample generated at its OWN coalescence time & phase, FD-whitened vs
the analysis PSD, weighted 5/95 band, no post-hoc alignment),
extract_ile_samples.py, dat_to_compact.py, make_reconstruct_subfile.sh
The ILE extrinsic run must use --fairdraw-extrinsic-output
--resample-time-marginalization so each sample carries its own coherent
coalescence time; the standard pipeline emits these via
--add-extrinsic-time-resampling.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Rift slowrot cubic
… is absent
In the in-loop-calmarg PILOT pipeline the wide ILE jobs and the last-iteration
EXTRINSIC ILE jobs are seeded from the previous iteration's consolidated cal
proposal, referenced as cal_consolidated_$(macroiterationprev).npz and (under OSG
file transfer) listed in condor transfer_input_files. When that seed file did not
exist on the submit node the job HARD-HELD (HoldReasonCode 13, "Transfer input
files failure ... No such file or directory") and the whole DAG deadlocked. Two
ways this happened:
1. The pilot builds MORE wide iterations than CALPILOT stages (e.g.
--calmarg-pilot-max-it 1 -> only cal_consolidated_0/1.npz produced, but wide
iterations 3,4,5 request cal_consolidated_2/3/4.npz, which are never made).
2. The last-iteration EXTRINSIC ILE node never set the macroiterationprev macro,
so cal_consolidated_$(macroiterationprev).npz expanded to the literal blank
cal_consolidated_.npz, which never exists.
The intended behavior is graceful degradation: an absent seed should fall back to
the prior placeholder (cal_consolidated_-1.npz, a valid prior breadcrumb always
created as the iteration-0 seed) instead of dead-holding. The ILE binary already
tolerates a missing/empty breadcrumb at run time (falls back to broad prior cal
draws), but condor transfers input files BEFORE the binary runs, so the hold
happens first -- a runtime-only guard cannot help. The fix therefore guarantees the
transfer SOURCE always exists, in DAG generation:
* bin/create_event_parameter_pipeline_BasicIteration: set macroiterationprev on
the last-iteration EXTRINSIC ILE node (it-1), matching the wide ILE node, so it
references a real per-iteration seed instead of the blank literal. This also
lets the extrinsic stage actually seed from the learned proposal on a shared FS
(previously the blank macro silently prevented that), and fixes the same blank
macro for the --extrinsic-handoff breadcrumb.
* bin/util_RIFT_pseudo_pipe.py: under --calmarg-pilot with OSG file transfer,
pre-seed a placeholder copy of the always-present cal_consolidated_-1.npz prior
breadcrumb for EVERY iteration index a wide or extrinsic job can reference
(0 .. n_iterations-1). A real calpilot OVERWRITES its placeholder at run time
via transfer_output_files (the DAG seed barrier guarantees ordering), so
behavior is unchanged whenever the learned seed IS produced; a missing seed now
degrades to the prior (placeholder proposal == prior -> zero-weight prior cal
draws) instead of holding. skip-if-exists preserves real seeds across a DAG
rescue/resume.
Why this approach: it is minimal, regression-safe (real seeds override the
placeholders, so present-seed behavior is byte-for-byte unchanged), and it fixes a
MISSING seed at the only layer that can prevent the condor transfer hold -- the
submit-node source. The shared-FS path already degraded gracefully (the breadcrumb
is not in transfer_input_files there, so the ILE runtime guard suffices); the
placeholder pre-seeding is scoped to the OSG file-transfer regime where the hold
occurs.
Adds RIFT/calmarg/test_seed_fallback.py locking the graceful-degradation contract:
(1) a prior-breadcrumb placeholder round-trips and, when seeded from, yields
UNWEIGHTED prior cal draws (importance log-weights == 0, i.e. placeholder == prior
fallback); (2) the ILE seed-present guard (os.path.exists and getsize > 0) and
breadcrumbs.load correctly classify a missing path and a 0-byte placeholder as
absent. All existing calmarg self-tests (adaptive, breadcrumbs, pilot,
test_cal_mc_error) still pass.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…struct_demo demo/reconstruct: whitened strain reconstruction with a CI band
…_graceful_degrade Rift o4d calmarg seed graceful degrade
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds dependency compatibility coverage for upstream dependency drift while keeping the GitLab belt-and-suspenders check in place.
Changes:
Evidence:
Related internal issues: oshaughnessy-junior/agent_internal_coding-rift#16, oshaughnessy-junior/agent_internal_coding-rift#17