Narrow national gender x age targets to five-year bands (unblocks publishing)

[vahid-ahmadi]

Fixes #441.

Problem

Publishing has been blocked since 19 June: every push.yaml run fails at test_population_fidelity.py::test_young_adult_population_not_collapsed, so the workflow never reaches make upload. No UK dataset has published for a month.

The national gender x age targets used 15-year bands:

0-14: 11.6M   15-29: 12.86M   30-44: 14.15M   45-59: 13.08M   60-74: 11.16M   75-90: 6.16M

18-24 sits entirely inside 15-29. A band that wide pins only its own total — calibration can satisfy 12.86M while distributing those people any way it likes inside the band, and it settles on ~3.4M in 18-24 (26% of the band) where ONS implies ~5.4M (42%). Total population passes at 69.1M because the missing 2M are absorbed into other bands; the error is compositional.

The finer 10-year bands do not help: all 108 ..._age_X_Y targets are REGION-level and constrain regional weights, not national ones. The national set had no age constraint narrower than 15 years.

Why this is the cause rather than a corrupting stage

Two builds with very different inputs converge on the same wrong answer:

Input	raw 18-24	final 18-24
Scrambled weights (pre-#436 published build)	2.98M	3.48M
Healthy weights (post-#436, rebuilt from source)	5.32M	~3.44M (CI)

A stage that corrupted the data would produce output tracking its input. Converging to ~3.4M from both 2.98M and 5.32M is what an unconstrained dimension looks like.

Change

Narrows _GENDER_BANDS to five-year bands. The ONS projection is published by single year of age and _aggregate_ages already sums arbitrary ranges, so this is a change of aggregation only — no new source data, no new download.

Also adds female_/male_ to _SKIP_PREFIXES for demographics.csv. That file hardcodes the superseded 15-year rows; without the skip they are emitted alongside the new five-year bands under a non-colliding naming scheme, double-counting the population. Verified: 138.6M before the skip, 69.0M after, against ONS ~69.5M.

Verification

After the change, 36 national targets across 18 contiguous five-year bands:

0-4 3.48M   5-9 3.93M   10-14 4.19M   15-19 4.19M   20-24 4.17M   25-29 4.50M
...
TOTAL 69.0M   (ONS ~69.5M)

20-24 is now pinned in its own band, and 18-24 straddles only two narrow bands instead of sitting inside one wide one.

Five tests pin the properties that matter: bands at most five years wide, contiguous and non-overlapping, 18-24 covered by more than one band, one target per band per sex (catching reintroduction of the CSV rows), and the national total in a sane range (catching double-counting).

What this does not claim

I have not run a full post-change build to confirm 18-24 lands near 5.4M — that needs CI. The prediction is explicit and falsifiable: if the guard still fails after this, the wide bands were not the binding constraint and the diagnosis in #441 is wrong.

Related

• Sort FRS household frame by ID to fix scrambled weights (2024-25 population undercount) #436 fixed a real scrambled-weights bug in the 2024-25 FRS ingestion, and is verified working (rebuilding raw from source restores corr(weight, size) to +0.226 and 18-24 to 5.32M). It has never been published, because the guard it added has blocked every release since.
• ni_class_2 / ni_class_4 OBR targets are also silently dropped on label-match failure, the same pattern Fix silently-dropped OBR receipts targets (sheet 3.9 is APD, not current receipts) #439 fixed for cash receipts. Not addressed here.