[codex] Cache PE SOI targets for rebuild PUF uprating#155
Conversation
|
Review note: I was unsure how aggressively to move SOI target handling away from |
|
Closing rather than merging — this targets code that no longer exists in an archived repo. microplex-us was converted to spec-only content in #261 (Jun 7-8) and then archived; For the record, the design was sound where it counts: the cache filename embeds the pinned policyengine-us-data commit so bumping the ref auto-invalidates, and the tests mock requests.get so there's no test-time network. Two things I'd have asked for were it live, both in soi.py: the cache write is non-atomic (write_bytes straight to the final path, so an interrupted download leaves a truncated file that header-only validation then accepts — temp file + os.replace fixes it), and header-only validation (nrows=0) won't catch a zero-row file. Worth carrying into whatever populace does for SOI sourcing. |
Fixes #147
Summary
--soi-pathas the explicit override for PE-SOI PUF uprating.microplex-usdownloads the pinned PE-style long SOI target table from thePolicyEngine/policyengine-us-datarepo into a versioned Microplex cache.policyengine_us_data/storage/soi.csvfile for SOI resolution while preserving repo-local PE-US-data handling for other PUF inputs, including raw PUF/demographics anduprating_factors.csv.soi.csv.Root Cause
The PE-SOI PUF uprating path previously expected
policyengine_us_data/storage/soi.csvwhen only--policyengine-us-data-repowas provided. Fresh PE-US-data checkouts do not reliably materialize that file, while the tracked historical PE-style target table lives atpolicyengine_us_data/storage/calibration_targets/soi_targets.csvin the PE-US-data repo.SOI Source
For this PR, the canonical default remains the PE-US-data SOI setup:
microplex-ususes the PE-style longsoi_targets.csvfrom a pinnedPolicyEngine/policyengine-us-datacommit and caches it assoi_targets_pe_us_data_<commit>.csv. This is a dependency on the PE-US-data repo artifact, not an import from the PE-US-data Python package.I checked the HF
policyengine-us-datamodel, but it does not currently publish the historical long SOI target table needed by this uprating path. It publishes current raw/target DB artifacts that do not cover the required 2015+ historical surface.Longer term, this should move to Arch: Arch should own source-backed IRS SOI facts/provenance across the needed years, while
microplex-usowns the adapter/export into the PE-style target surface used by PUF uprating.Validation
uv run --no-sync pytest -q tests/test_puf_source_provider.py -k "soi or download_pe_soi"-> 8 passed, 29 deselecteduv run --no-sync pytest -q tests/test_puf_source_provider.py tests/pipelines/test_pe_us_data_rebuild.py tests/pipelines/test_pe_us_data_rebuild_checkpoint.py::test_main_passes_donor_condition_selection_override-> 47 passeduv run --no-sync ruff check src/microplex_us/data_sources/puf.py tests/test_puf_source_provider.pyuv run --no-sync python -m py_compile src/microplex_us/data_sources/puf.py tests/test_puf_source_provider.pygit diff --checkNotes
rgcaller scan for the touched PUF/rebuild symbols before committing.