Reduce CGT band donors to 30 per band for L0 sparsity economics#445
Merged
Conversation
The first full 512-epoch build with 300 donors per band passed the taxpayer-count outcome test but failed total gains (£20.7bn vs £65.9bn) and concentration (20% vs ~61% from £1m+): the L0-regularised calibration pays a gate penalty per active record, and keeping 2,700 light donors (weight ~7 each in the top band) open cost more than the band targets recovered, so donor weight shrank ~8x and the £2m+ bands landed 94-97% under target. The outcome tests gated the upload, so nothing was published. Thirty donors per band carry the same band-exact initial totals with a tenth of the gate cost. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The first full build after #444 was correctly blocked from uploading by the built-dataset outcome tests: the taxpayer count passed (532k, down from 1.86m), but total above-AEA gains came out at £20.7bn vs HMRC's £65.9bn and only 20% of gains from £1m+ gainers vs ~61%. The calibration logs show why: all 2,700 donors kept positive weight but their total weight shrank ~8x (139k prior → 16.6k), leaving every band target 32-97% under, worst at the top (£2m band -94%, £5m band -97%).
Cause
The L0-regularised calibration pays a HardConcrete gate penalty per active record. A top band represented by 300 rows of weight ~7 each is expensive to keep open relative to the loss it recovers on its two band targets, so the optimiser shrank the donors and accepted the CGT undershoot.
Fix
30 donors per band instead of 300: identical band-exact initial totals (top band: ~67 weight each instead of ~6.7), a tenth of the gate cost. All structural tests derive from
DONORS_PER_BANDand pass unchanged.If the full build still undershoots the top bands after this, the next lever is exempting donor rows from L0 gating or group-weighting the CGT targets — but this is the cheapest change that directly attacks the observed failure mode.
🤖 Generated with Claude Code