Require provenance_class (measurement basis) on every record set and fact#96
Open
MaxGhenis wants to merge 1 commit into
Open
Require provenance_class (measurement basis) on every record set and fact#96MaxGhenis wants to merge 1 commit into
MaxGhenis wants to merge 1 commit into
Conversation
Populace calibrates against these facts, and the standing design rule is: never calibrate against statistics that are functions of the latent quantities populace corrects; admin data first, survey aggregates only for items surveys measure well. The ledger stays epistemically neutral and ingests everything — the measurement-basis classification becomes a required schema field so consumers can gate on it fail-closed. Every record set and every emitted fact now carries provenance_class, closed to exactly administrative, census, survey_aggregate, and model_output, with no default; survey_aggregate requires a non-empty survey_instrument sibling that is forbidden for the other classes. Validation runs at build and at load, the field flows through record-set specs, source records, aggregate facts, fact and consumer-contract JSONL, consumer artifact loading and resolution, and the SQLite artifact, and a completeness test prevents any future record set from omitting it. The field joins the record-set spec hash but is deliberately excluded from stable fact identities, so fact keys are unchanged. All 67 existing packages (1,514 record sets) are backfilled with per- package justifications recorded in the implementation handoff: tax and benefit program records (SOI, SNAP, Medicaid/Medicare, TANF, LIHEAP, ACA, CBO historical actuals, HMRC VAT, state tax collections) are administrative; census-controlled PEP counts are census; ACS tabulations are survey_aggregate/ACS 1-year and the ICI fact-book table is survey_aggregate/ICI Monthly Long-Term Survey; BEA NIPA and Regional accounts, Census population projections, CBO baseline projections, JCT tax-expenditure estimates, and Federal Reserve Z.1 net worth are model_output — assembled statistical estimates, not source records. Implemented by GPT-5.6 Sol with a per-classification justification table; reviewed line by line, with one reclassification at integration (Z.1 administrative → model_output for consistency with the NIPA principle). 95 tests pass including the malformed build/load matrix. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
MaxGhenis
force-pushed
the
provenance-class
branch
from
July 21, 2026 09:07
ff20a50 to
16a6b7b
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds the required, closed-vocabulary
provenance_classfield (administrative | census | survey_aggregate | model_output; survey_aggregate additionally requiressurvey_instrument) with no defaults, validated at build and load, flowing end to end through specs, facts, consumer contract, artifact load, and SQLite. All 67 existing packages / 1,514 record sets are backfilled with per-package justifications (see commit message; full table in the implementation handoff). Fact identities are unchanged; record-set spec hashes change (the field joins the spec hash).Purpose: populace's calibration gate will default to administrative+census targets only, with survey aggregates requiring reviewed opt-in and model outputs never fitted (evaluation/oracle use only) — so nothing populace is built to correct becomes a fitting target.
Sequencing note: overlaps with #87 on
policyengine_ledger/consumer.py(and #87 packages its own consumer schema copy that predates this field). Recommended order: merge #87 first (green + mergeable since 7/10), then I rebase this and reconcile the packaged schema; merging this first forces #87 to rebase instead. Either works — flagging so neither lands blind.🤖 Generated with Claude Code