Benchmark

Frozen 20-task suite for Otter's local Qwen 7B coding path. Not a general model leaderboard. The headline number is end-to-end implement success — a valid patch that applies, stays on expected files, and passes or skips gold tests.

Latest run is v0.6 on qwen2.5-coder:7b. It is better than v0.5 and it is not done. Retrieval and planning are heuristic Otter stages; only generation uses the LLM.

Headline (v0.6)

20 tasks / 4 repositories (sample-app, bottle, click, starlette). 9 locate, 11 implement. Unexpected files stayed at 0% — quality gates were not weakened to raise E2E.

What this measures

• Retrieval Recall@K / Precision@K against gold files
• Plan grounding (deterministic overlap, not a human judge)
• Structured patch JSON, apply, expected-file accuracy
• Syntax + targeted tests after apply
• Wall-clock latency

It does not measure time-to-merge, cloud models, MCP/CLI UX, or human preference. Auto-approve exists only inside benchmarks/workspaces/.

v0.6 still fails

• 6 test failures — the patch applied, then gold tests (or expect-in-files) failed. Qwen 7B can emit valid JSON and still write the wrong test or API.
• 1 incomplete_auth — the quality gate correctly rejected a login change with no login/register route.
• JSON malformed, destructive rewrites, missing edit targets, and syntax failures from v0.5 are gone.

Reproduce

# Ollama must already have qwen2.5-coder:7b (the suite never pulls)
python -m venv benchmarks/.venv
benchmarks/.venv/bin/python -m pip install -r apps/api/requirements.txt -r benchmarks/requirements.txt
export PYTHONPATH="$PWD:$PWD/apps/api"
benchmarks/.venv/bin/python -m benchmarks.runners.run_benchmark --model qwen2.5-coder:7b

Windows uses benchmarks\.venv\Scripts\python.exe and set PYTHONPATH=%CD%;%CD%\apps\api. Full notes: benchmarks/README.md.

Reports

• qwen-v0.6-report.md
• Models — local Ollama setup used for this suite