Test the code examples in your documentation — automatically, in CI.
The examples in a project's README are the first thing humans and AI assistants copy, and almost nobody tests them after writing them. The software changes, the examples don't, and they quietly break. readme-ci finds the code snippets in your Markdown documentation and (soon) executes them in a sealed sandbox, so "our examples always run" can be a CI-enforced promise instead of a hope.
v0.0.1 in progress — early and honest about it.
| Stage | Status |
|---|---|
| Extract fenced code snippets from Markdown | ✅ works |
| Classify snippets (runnable / placeholder / unsafe / needs-network) | ✅ heuristic v0 |
| Execute snippets in a network-isolated Docker sandbox | ✅ implemented |
| Pass/fail report + CI gate | ✅ readme-ci run |
readme-ci scan only reads your docs (no execution). readme-ci run executes the
snippets classified as runnable — but only inside a locked-down Docker sandbox.
When no sandbox is available, runnable snippets are reported as skipped: readme-ci
never falls back to running untrusted code on the host.
python3 -m venv .venv && source .venv/bin/activate
pip install -e ".[dev]"
readme-ci scan README.mdscan lists every fenced code block it finds, with language, line numbers, and a conservative classification:
README.md:22-26 [bash] 3 line(s) needs-network
To actually execute the runnable examples (in a sandbox) and get a pass/fail report suitable for CI:
readme-ci run README.md
readme-ci run docs/ --timeout 20 --jsonrun exits non-zero if any snippet fails, so it works as a CI gate. Add
--json to either command for machine-readable output.
An example that shouldn't be executed — it needs a real service, is purely
illustrative, or is intentionally broken — can be marked so readme-ci run
leaves it alone, without adding noise to the code readers copy. Either:
- put
readme-ci:skipin the fence info string, right after the language (e.g. an opening fence of```python readme-ci:skip), or - put
<!-- readme-ci:skip -->on the line immediately above the fence.
The marker never appears in the copyable code, and the snippet is reported as
directive-skip instead of being run.
Documentation snippets are untrusted input. When execution lands, it will be:
- Sandboxed — containers with networking disabled and strict timeouts
- Conservative — snippets containing destructive patterns (
rm,sudo, pipe-to-shell, credential placeholders likeYOUR_API_KEY) are classified as non-runnable and skipped - Opt-in — nothing runs outside the sandbox, ever, and nothing runs at all unless you ask
readme-ci is the first tool from machine-operable, a research program measuring — and fixing — how reliably open source repositories work for the humans and AI agents that depend on them. Broken documentation examples don't just mislead newcomers anymore; they get learned and repeated by AI assistants at scale.