Loop Engineering — a minimal, runnable demo

Loop engineering is writing the program that prompts your agent instead of prompting it yourself, turn by turn. You give the loop a goal and a way to check it; the loop makes the agent act, observe real results, and revise until the goal is verifiably met — or a guardrail stops it.

"I don't prompt Claude anymore. I have loops running that prompt Claude... My job is to write loops." — Boris Cherny, Anthropic

This repo is ~200 lines that make the idea concrete, driven by a local model (qwen3-coder:30b via Ollama) on a single RTX 3090. No cloud, no framework.

The three ideas it demonstrates

1. Ground-truth feedback. Success is decided by pytest's exit code, not by the model's opinion of its own work. The test suite is the thing in the loop that can say no.
2. The maker/checker split. The model that writes the code (qwen3-coder:30b) is not the one that grades it. A second, different model (qwen3:14b) reviews adversarially — because a model grading its own homework is far too generous.
3. Bounded autonomy. A real loop needs more than act -> observe -> repeat. This one has four explicit exits:

   Exit	Meaning
   SUCCESS	tests pass — the goal is verifiably met
   BLOCKED	the agent declares it cannot proceed
   CAPPED	hit the hard iteration limit (the dumb backstop)
   STUCK	no-progress detection: the same failure recurred, so stop re-trying a doomed fix

Setup

python3 -m venv .venv
.venv/bin/pip install -r requirements.txt
ollama pull qwen3-coder:30b   # the maker (MoE, ~3.3B active, fits 24GB)
ollama pull qwen3:14b         # the checker (a *different* model)

Run it

# 1. The happy path: the loop fixes buggy Roman-numeral code until tests pass,
#    then stops on its own. Add --checker for the adversarial review.
.venv/bin/python loop.py solvable --checker

# 2. The guardrail in action: an impossible task. No-progress detection bails
#    after the failure repeats, instead of thrashing.
.venv/bin/python loop.py unsolvable

# 3. THE ABLATION (the teaching moment): same impossible task, but with
#    no-progress detection turned OFF. Watch it waste its ENTIRE budget
#    re-trying the same doomed fix before hitting the hard cap.
.venv/bin/python loop.py unsolvable --no-guardrails

Run #2 vs #3 side by side is the whole lesson: designing the loop is half of it; the other half is putting something in it that can say no — and knowing when to quit.

v2: the same loop, building a website

web/web_loop.py applies the identical four-exit skeleton to a real task — building a styled landing page — by swapping the single pytest oracle for a ladder of checkers:

maker (qwen3-coder:30b) writes site/index.html
        │
        ▼
[Tier 0-2] deterministic_check   real headless chromium: structure (h1, nav,
        │                        sections, footer), no broken anchors, no
        │  (passes)              console errors. Ground truth.
        ▼
[Tier 3]  vision_check           qwen2.5vl screenshots the page and grades
                                 whether it LOOKS finished and styled.

The cheap deterministic gate runs first; the expensive vision check only runs on a page that already cleared it. A design lesson baked in: no-progress detection (STUCK) keys only off the deterministic tier — the vision verdict is noisy, so it's bounded by the iteration cap instead. You can only reliably detect "no progress" against a stable oracle.

.venv/bin/pip install -r requirements.txt
.venv/bin/python -m playwright install chromium   # one-time browser download
ollama pull qwen2.5vl:7b                           # the vision checker

.venv/bin/python web/web_loop.py                  # build the page
.venv/bin/python web/web_loop.py --no-vision      # deterministic tier only

On a typical run the loop fixes the page's structure on iteration 1 and passes both tiers on iteration 2, then stops. See web/SPEC.md for the goal it builds toward.

Files

File	Role
loop.py	the engineered loop: four exits, maker/checker, no-progress detection
agent.py	thin Ollama client (stdlib only)
VISION.md	anchor file — the goal, on disk, where the loop can't forget it
tasks/solvable/	buggy Roman-numeral code + a fixed test oracle
tasks/unsolvable/	contradictory tests — unsatisfiable on purpose
web/web_loop.py	v2: the loop building a website, with a deterministic + vision checker ladder
web/checks.py	Playwright deterministic checker + qwen2.5vl vision checker
web/SPEC.md	the site spec — anchor file for the web loop

*.seed.py files are the pristine buggy starting points; the loop copies them to the working file at the start of every run, so runs are repeatable.

License

MIT.