Fit a Pareto upper tail to the capital gains imputation

[vahid-ahmadi]

Measured before/after

The bug is visible on the published enhanced_frs_2024_25.h5 (2026). Reading that file directly:

Statistic	Published model	HMRC
Share of gains from gains >= £1m	1.6%	~61%
Largest single gain	£1.92m	HMRC top band is "£5m and over"
Mean gain per CGT taxpayer	£38,181	~£175,000
CGT taxpayers (> £3k AEA)	1.839m	~359k
Total gains	£70.8bn	~£63bn
CGT revenue	£14.0bn	OBR £20.8bn

To isolate this change I ran impute_capital_gains standalone on that dataset with only the sampler swapped (same pipeline, same seed, both arms):

Statistic	Before	After	HMRC
Share of gains >= £1m (taxpayers)	20.9%	48.5%	~61%
Share of gains >= £5m (taxpayers)	0.0%	29.8%	~36%
Largest single gain	£1.98m	£275.6m	£5m+ band exists
Median gain per taxpayer	£27,566	£27,566	~£25,000
Mean gain per taxpayer	£77,844	£124,191	~£175,000
CGT taxpayers (> £3k AEA)	1.224m	1.224m	~359k
Total gains	£95.4bn	£152.1bn	~£63bn
CGT revenue	£21.2bn	£34.8bn	OBR £20.8bn

What changed

UnivariateSpline over [0.05 … 0.95] with k=1, sampled at cg_rng.random(), linearly extrapolates every draw above p95 off the last segment. The heavy tail of capital gains is simply absent.

Gains above p95 are now drawn from a Pareto tail anchored at each income band's p95. The shape parameter is fitted, not guessed. The same Advani & Summers table already publishes mean_gains_given_gains per income band — a column the imputation never used — and the spline alone undershoots it by 35–50% in every band. That gap is the missing tail mass, so for each band we solve (Brent, on a closed-form capped-Pareto mean) for the shape parameter making body + tail reproduce the published band mean. Fitted values cluster at 1.15–2.07, median 1.47, which is where the CGT literature puts this tail.

Because the fit is per income band, the income-band conditioning is preserved for free rather than a single global tail being imposed. The spline body below p95 is bit-for-bit unchanged (test-pinned), as is the seeded generator and the number of draws consumed.

What this does not fix — please read

• Taxpayer count is untouched. 1.224m before and after. This change alters gain amounts, not who has a gain; the count is set by percent_with_gains and the weight-blending step. HMRC's ~359k is still far off and is not addressable here.
• Levels are not calibrated in these numbers. The standalone run skips the downstream reweighting, so total gains and revenue overshoot in both arms (the "before" arm already shows £95.4bn vs the published £70.8bn). The meaningful comparison in the table above is the concentration and median rows, which are scale-invariant. Setting the level is the reweighting step's job — and it now has hmrc/capital_gains_total and hmrc/cgt_taxpayers to do it with. The point of this PR is that reweighting could never have fixed concentration, only the level. A full pipeline rebuild is still needed to confirm the calibrated end state; I did not run one.
• Concentration lands at ~48% vs HMRC's ~61%. The fit targets Advani & Summers band means, whose population (everyone with gains) is broader than HMRC's (CGT-liable taxpayers only), so it should be expected to land below. I chose to stay anchored to A&S rather than fit the tail directly to HMRC, because A&S is the source the body comes from and mixing the two populations in one distribution would be incoherent. This is the deliberate trade-off.
• One income band has an anomalously low p95 relative to its published mean and solves to alpha ≈ 1.0; it is floored at 1.15 and consequently undershoots its band mean. Tail draws are capped at £500m.

Reference data

HMRC CGT statistics Table 2.1a (2023-24, individuals) is committed as storage/capital_gains_size_distribution_hmrc.csv. The .ods asset URL embeds a content hash that changes each release, so it is not a stable fetch target; this follows the ons_demographics.py precedent. Note HMRC's figures "only include taxpayers who have a CGT liability" — a narrower population than the microdata's, documented in the file header.

Tests

policyengine_uk_data/tests/test_capital_gains_tail.py (9 tests) pins: body below p95 identical to the original spline; tail anchored at p95, monotone and capped; band means reproduced; determinism under the seed; fitted alphas in a plausible range; concentration delivered; and a guard test asserting the body alone undershoots the published means, so the justification for the tail can't silently rot.

Incidental finding: the "spline" uses scipy's default smoothing s, so it never interpolated the published percentiles exactly. Pre-existing; preserved, not changed.

🤖 Generated with Claude Code

[vahid-ahmadi]

Superseded by #444, which addresses the same missing upper tail with existing repo patterns instead of a fitted Pareto: zero-weight donor households at each HMRC Table 2.1a band mean, weighted by calibration against per-band count and gains targets. With the band structure targeted directly, the parametric above-p95 shape assumption is no longer needed, and merging both would double-correct the tail. The HMRC Table 2.1a reference CSV introduced here is carried over into #444.