perf: incremental candidate counting for disconnected training

[AmitMY]

The training loop rebuilt Counter(graph.get_merges()) from scratch every step — re-counting and re-hashing every candidate in the whole forest, even the subgraphs the chosen merge never touched. Profiling showed this recount is ~70% of training time. For a disconnected forest (BPE/BNE — each merge hits only the few words containing the pair) almost all of it is wasted.

_train_incremental keeps each subgraph's candidate counts plus a running global total, and after a merge updates only the subgraphs that contained it (located via an index):

for i in list(index[nodes]):
    _index_remove(total, index, i, comp_counts[i])      # subtract old
    components[i] = components[i].merge(token, nodes)
    comp_counts[i] = Counter(components[i].get_merges())
    _index_add(total, index, i, comp_counts[i])          # add new

Correctness: total is summed in subgraph order, so picking the first max-score candidate reproduces max(Counter(graph.get_merges()), key=score) exactly, including tie-breaks. Output is byte-identical — merge digests unchanged, 138 tests pass.

Gated to the disconnected case (Tokenizer passes incremental=not connected); connected / draw / verbose keep the plain recount loop, so Boundless and SuperBPE are unchanged (incremental doesn't help when a merge touches most of the few large components).

	main	this PR
BPE	0.384s / 1.33MB	0.091s / 4.21MB — 4.2×, +mem
BNE n=4	0.768s / 2.63MB	0.205s / 8.93MB — 3.7×, +mem
Boundless	0.469s / 1.42MB	0.464s / 1.42MB — unchanged

Trade-off: the persistent per-subgraph count state (comp_counts + index + total) raises peak memory ~3× for the disconnected tokenizers — a deliberate speed-for-memory choice for the dominant BPE/BNE path.

🤖 Generated with Claude Code