DepthBench-LC

Long-Context Operating Point Suite

DepthBench-LC is a lane-based benchmark suite for characterizing local LLMs at filled context depth across fit, prompt ingress, decode, retrieval, and coherence. It is meant to sit above any one subtest, so results stay grounded in the model's actual long-context operating point instead of raw throughput or KV compression in isolation.

Core suite modules:

• DB-LC/fit: maximum passing context and OOM boundary
• DB-LC/pp: prompt-processing throughput, PP(L)
• DB-LC/tg: decode throughput at actual filled depth, TG(L)
• DB-LC/niah: exact and semantic long-context retrieval
• DB-LC/memo: memo-base and memo-at-depth coherence probes

Naming:

• Canonical suite name: DepthBench-LC
• Formal doc title: DepthBench-LC: Long-Context Operating Point Suite
• Repo / CLI: depthbench
• Report prefix: DB-LC v1.0

Example report title:

DB-LC v1.0 | Nemotron-Cascade-2-30B | GB10 | IQ4_XS | q8_0 KV

Repo Layout

• docs/specification.md: suite standard and execution protocol
• docs/agent-contract.md: formal agent interface — required inputs, output schemas, execution invariants, failure protocol
• docs/agent-checklist.md: non-negotiable checklist for agent execution
• docs/prompt-pack.md: orchestration, judge, dataset, and report prompts
• docs/report-template.md: report skeleton matching the standard
• config/example.toml: example run configuration (GB10 / high-memory)
• config/consumer_gpu.toml: example run configuration (RTX 3090 / RTX 4090, 24 GB VRAM)
• runs/example_golden/: golden reference run with synthetic but schema-valid artifacts
• src/depthbench/cli.py: depthbench CLI for title generation, module listing, run scaffolding, and run validation

Quick Start

python3 -m venv .venv
. .venv/bin/activate
python -m pip install -e .
depthbench modules
depthbench report-title \
  --model Nemotron-Cascade-2-30B \
  --hardware GB10 \
  --quantization IQ4_XS \
  --kv-label "q8_0 KV"
depthbench scaffold-run \
  --config config/example.toml \
  --output-dir runs/example_nemotron
depthbench validate --run-dir runs/example_golden

Agent-driven execution

DepthBench-LC is designed as an Agent-Run Benchmark Specification (ARBS). The suite defines the contract; an automation agent (Cortex, Codex, Claude, or custom scripts) executes it.

The agent workflow:

1. depthbench scaffold-run creates the run directory and manifest
2. The agent follows docs/agent-checklist.md to execute each module
3. depthbench validate --run-dir ... verifies the run meets the contract
4. The agent loops on validation failures until the run passes

See docs/agent-contract.md for the full interface specification including JSON schemas, execution invariants, and failure handling rules.

Status

This repository is the initial packaged release of the suite standard and prompt pack. The CLI standardizes naming, module metadata, report titles, run-directory scaffolding, and post-run validation so benchmark runs start from the same structure and can be verified against the agent contract.