Skip to content

feat(core): model-architecture seam, chat templates, text antiprompt - #667

Open
khalilswdp wants to merge 1 commit into
JustVugg:devfrom
khalilswdp:feat/arch-seam
Open

feat(core): model-architecture seam, chat templates, text antiprompt#667
khalilswdp wants to merge 1 commit into
JustVugg:devfrom
khalilswdp:feat/arch-seam

Conversation

@khalilswdp

@khalilswdp khalilswdp commented Jul 28, 2026

Copy link
Copy Markdown
Contributor

Summary

The engine is hardcoded to GLM-5.2: chat template, eos token, and forward-pass dispatch are all baked into colibri.c, so a second model family currently means a second binary (like olmoe.c). The smallest change that fixes this is a load-time seam:

  • arch.h + arch_glm.c β€” a ModelArch descriptor registry selected from config.json "architectures"[0] (falling back to "model_type"). Unknown architectures are refused up front with the supported list, instead of silently mis-reading another model's weights. A config with neither field predates the registry and keeps loading as GLM with a notice, so existing snapshots and oracle fixtures are unaffected.

  • engine.h β€” the shared engine types (Cfg, QT, Layer, Model, …) move verbatim out of colibri.c so a future arch module can compile as a standalone .c. The forward pass dispatches attention_rows/moe through a two-pointer vtable bound at load; GLM binds the existing built-ins. No speculative fields β€” a new family adds its fields in the PR that consumes them.

  • Chat template/eos from the descriptor β€” serve/run build turns from chat_prefix/chat_turn/chat_think/chat_eos instead of hardcoded GLM strings; the GLM descriptor is byte-identical to the previous literals. router_bias gains NULL guards for families without e_score_correction_bias.

  • antiprompt.h β€” a quantized model can emit a role delimiter like <|user|> as ordinary text tokens, so the id stop never fires (and serve drops role-marker ids from the stop set, [Bug]: Tool Calling Fail in serve APIΒ #401) and the model hallucinates the next turn. This matches the decoded text, streamed with a (longest-needle βˆ’ 1) hold-back so delimiters split across tokens are caught and never printed. COLI_ANTIPROMPT overrides or disables it.

Also: COLI_TPS_EVERY makes the tok/s heartbeat cadence configurable (default 8; the hardcoded 16 meant short replies never produced a reading; 0 silences).

Validation

  • make -C c check β€” clean build, 0 new warnings, C unit tests + Python
    stdlib tests pass (the two -Wformat-truncation notes gcc 15 emits are
    pre-existing on dev; a separate small PR fixes them)
  • CUDA changes β€” n/a (no kernel changes; arch_glm.o is pure C linked into
    every build)
  • No performance claims in this PR

Compatibility

  • The default CPU build remains dependency-free β€” the registry is pure C,
    two new headers + one .c, no new deps
  • No model files, generated binaries, or benchmark artifacts included
  • Defaults changed: none. All new behavior is selected by the model's own
    config.json or opt-in env vars (COLI_ANTIPROMPT, COLI_TPS_EVERY).
  • .gitignore gains pattern rules for build outputs (*.o, *.obj, …) so new
    sources' objects don't leak into git status.

@JustVugg

Copy link
Copy Markdown
Owner

Thanks for this β€” the architecture seam is a nice direction. It now conflicts on dev (colibri.c / Makefile moved with recent merges). Could you rebase onto latest dev? CI is green; once it's clean we can land it, then #670 on top of it.

@JustVugg JustVugg added the enhancement New feature or request label Jul 28, 2026
@khalilswdp
khalilswdp force-pushed the feat/arch-seam branch 2 times, most recently from 4843afe to 074b9a8 Compare July 28, 2026 21:24
@khalilswdp

Copy link
Copy Markdown
Contributor Author

Rebased onto latest dev (a3a5a75) β€” applied cleanly, no conflicts. The conflicts you saw were from an accidental fork-sync merge that briefly polluted the branch; it's been removed and the branch is back to the single seam commit, now based on current dev. Ready to land β€” and the windows suite is already re-stacked on top for the follow-up.

@khalilswdp

Copy link
Copy Markdown
Contributor Author

Rebased onto latest dev (c67dbab). One real conflict with #679: both changes guard the spec-decode emit loop β€” your g_mux_stop/g_mux_cancel (client STOP/CANCEL mid-turn) and this PR's g_astop (text antiprompt) β€” resolved as the union of the stop conditions so both features coexist.

Why the antiprompt guard exists at all: a heavily-quantized GLM can emit a role delimiter like <|user|> as ordinary text tokens (<,|,user,|,>) instead of the atomic control-token id β€” the id stop never fires and the model runs on, hallucinating the next turn. And in serve mode the role-marker ids are deliberately dropped from the stop set anyway (#401, protecting <tool_call> blocks from int4 argmax noise), so even the atomic token wouldn't stop there. The antiprompt matches the decoded text instead, streaming with a short hold-back (longest needle βˆ’ 1) so a delimiter split across tokens is caught and never leaked to the client.

Full local gate green at the rebased tip: build + C/Python suites, token-exact oracle 32/32 TF / 20/20 greedy, no new warnings. Ready to land β€” the windows suite is re-stacked on top and ready to follow.

@JustVugg

Copy link
Copy Markdown
Owner

dev has moved a fair amount (v1.3.0: Kimi K3 and Inkling engines, the ragged-attention grouped-scale fix, CUDA_RELEASE_HOST default, the Inkling expert-cache fix). This PR now conflicts β€” could you rebase onto latest dev? Nothing is wrong with the change itself; CI was green where it ran. Ping me once it's clean and I'll review/merge.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request needs-rebase Confligge, serve rebase dell'autore

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants