Commit EFO workbooks as fallback for OBR downloads#446
Merged
Conversation
obr.uk serves 403 Forbidden to GitHub Actions runner IPs (every attempt on the 2026-07-21 push builds 403'd while the same URLs work from residential connections). The swallowed error silently dropped 28 OBR targets, so the build calibrated without income tax, NI, VAT or CGT-tax targets and failed the reform-impact and registry tests, blocking the data upload. Commit the receipts and expenditure workbooks for the vintage pinned in sources.yaml under storage/obr_efo/ (following the committed-not-fetched precedent of capital_gains_size_distribution_hmrc.csv and ons_demographics.py) and fall back to them whenever the download ultimately fails. Permanent HTTP errors (403) no longer burn the retry budget. Tests pin that the full 34-target OBR set builds fully offline. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Both full builds on main after #445 failed the same way: obr.uk returns 403 Forbidden to GitHub Actions runner IPs (confirmed in the job logs; the same URLs work from residential connections). The error is swallowed by design, so 28 of 34 OBR targets silently vanished, the calibration ran without income tax / NI / VAT / CGT-tax targets, and the degraded weights failed
test_reform_impacts(VAT +2pp: £41.9bn vs £31.3bn) and the OBR registry tests — correctly blocking the data upload. Meanwhile the CGT band targets from #444/#445 all landed within 5-12% on the full build, so this 403 is the only thing between main and a publishable dataset.Fix
sources.yamlunderstorage/obr_efo/, following the committed-not-fetched precedent of the HMRC size-of-gain table andons_demographics.py._download_workbookstill tries the live URL first (with the existing 429/5xx retry), but falls back to the committed workbook when the download ultimately fails; permanent errors like 403 no longer burn the retry budget.sources.yamlURLs on each EFO vintage change.Tests
test_obr_efo_fallback.py: committed workbooks exist and parse (receipts sheet lookup works), 403 falls back without retrying, connection failures fall back, unknown URLs still raise, and the full 34-target OBR set builds fully offline.🤖 Generated with Claude Code