Compartments marked as `legacy=1` despite Historian V2 producing correct tier tags — `extractTier` regex fails on model output with mismatched closing tags

[null-axiom]

What happened?

Compartments are being flagged as legacy despite the current Historian V2 producing <p1>/<p2>/<p3>/<p4> tier tags in the output. The "Historian V2 is released — upgrade your compartments" message appears for sessions where the historian ran successfully with legacy=0 on the run itself.

Root cause: The tier extraction parser (extractTier / TIER_REGEXES) fails silently when the historian model produces malformed closing tags. The model closes <p1> with </p2> instead of </p1>, so the regex /<p1\s*>(.*?)<\/p1>/s doesn't match. When p1 extraction fails, the compartment falls through to the legacy code path and is marked legacy=1.

---

Evidence:

1. DB stats (1,042 compartments total):

• 73 compartments with legacy=1 (all have p1=NULL)
• 969 compartments with legacy=0 (all have p1 populated)
• Zero non-legacy compartments have NULL p1

2. All 73 legacy compartments contain tier tags in content but NOT in the p1 column:

Check	Legacy (73)	Non-legacy (969)
<p1> opening tag in content	✅ YES	N/A (tags extracted, content is clean text)
</p1> closing tag in content	❌ NO	N/A
</p2> closing tag in content	✅ YES	N/A
p1 column	NULL	Populated

3. The model output looks like this:

<compartment start="1" end="50" title="...">
  <p1>
  ...content...
  </p2>          ← wrong closing tag — should be </p1>
  <p2>
  ...content...
  </p2>
  <p3>...</p3>
  <p4>...</p4>
</compartment>

4. historian_runs.legacy = 0 for ALL runs that produced legacy compartments:

This is the disconnect — the run is NOT marked legacy, but the compartments it produced ARE. The legacy flag on compartments is derived from whether p1 was extracted, not from the run itself.

---

Code reference:

extractTier and TIER_REGEXES (in compartment-parser.ts / dist index.js):

function makeTierRegex(n) {
  return new RegExp(`<p${n}\\s*/>|<p${n}\\s*>(.*?)</p${n}>`, "s");
}

function extractTier(inner, index) {
  const m = inner.match(TIER_REGEXES[index]);
  if (!m) return;  // undefined — regex didn't match
  return unescapeXml2((m[1] ?? "").trim());
}

The regex /<p1\s*>(.*?)<\/p1>/s requires an exact </p1> closing tag. When the model produces </p2> as the closing tag for <p1>, the regex doesn't match → extractTier returns undefined → p1 is not set.

insertCompartmentRows (in compartment-storage.ts):

function insertCompartmentRows(db, sessionId, compartments, now) {
  for (const compartment of compartments) {
    const hasTiers = typeof compartment.p1 === "string" && compartment.p1.length > 0;
    stmt.run(..., hasTiers ? 0 : 1, ...);  // legacy = hasTiers ? 0 : 1
  }
}

When p1 is undefined (extraction failed), hasTiers is false → legacy = 1.

---

The chain:

Model outputs <p1>...</p2> (mismatched closing tag)
  → extractTier regex /<p1>(.*?)<\/p1>/s doesn't match
  → p1 = undefined
  → parseCompartmentOutput falls through to fallback (no p1/p2/p3/p4 fields)
  → insertCompartmentRows: hasTiers = false
  → legacy = 1
  → Dashboard shows "legacy" tag
  → "Historian V2 upgrade" message appears (false positive)

---

Model context:

Historian model: opencode/deepseek-v4-flash-free. The model consistently produces <p1>...</p2> instead of <p1>...</p1>. This has been observed across 73 compartments spanning from April 28 to July 23, 2026 — including compartments created today.

---

Possible fixes:

1. Parser-side: Make the tier regex more lenient — e.g., /<p${n}\s*>(.*?)<\/p[1-4]>/s to accept any </pN> closing tag, then validate the closing tag number matches
2. Parser-side: Add a fallback that tries /<p1\s*>(.*?)<\/p\d+>/s when the exact match fails
3. Prompt-side: Add explicit instruction in the historian prompt to verify closing tags match opening tags

---

Environment:

• Magic Context: v0.32.4
• Historian model: opencode/deepseek-v4-flash-free
• Historian disallowed_tools: ["*"] (though legacy compartments existed before this was set)
• DB: ~/.local/share/cortexkit/magic-context/context.db

[alfonso-magic-context]

Confirmed — this is one of the best-diagnosed reports we've received, and your chain is exactly right: the tier regex requires the exact </pN> close, deepseek-v4-flash emits </p2> for <p1>, extraction returns undefined, the parser falls through to the v1-flat path, and insertCompartmentRows derives legacy=1 while the run-level flag stays 0. The disconnect between the two flags is real.

One thing that changed since v0.32.4: v0.33.0 (released today) made tier validation strict — tierless parses now fail into retry-with-feedback and model fallback instead of silently landing legacy rows. On your model that would turn silent legacy rows into retries, which is better but still wrong for output that's 99% valid with one mangled close tag.

The fix we're shipping combines your options 1 and 3, plus healing for your stranded rows:

1. Lenient close matching: an opened <pN> now terminates at the next </p\d> regardless of digit (with an over-capture guard so a tier can never swallow the next tier's opening tag) — in both the TypeScript parser and its Rust twin.
2. Prompt hardening: the historian prompt now explicitly requires matching closing tags.
3. Heal migration: your 73 legacy rows still carry the full tier content in content — the next release re-parses them with the lenient extractor, populates p1–p4, and clears the legacy flag. The false upgrade nag disappears retroactively; no recomp needed.

Will update here when it ships.