Updated October 20, 2026
Published coding-agent studies
Inspect measured results, failed and unknown execution, study limits and prior publication snapshots without an account.
Adversarial TDD skill: supplied-function tasks
October 10 cohort · completed execution · formal verdict inconclusive · publication revision 5. Eight synthetic families, 32 attempts. All 16 attempts without the skill passed, using 3.41 million tokens versus 8.39 million with it. All 16 attempts with the skill passed too. With the skill, total tokens were 2.46 times as high and summed elapsed time was 1.60 times as long as without it. One strict read qualification remains.
Adversarial TDD skill: repository pilot
October 12 · completed 18-attempt schedule across two registered execution windows · exploratory/inconclusive · publication revision 2, protocol revision 2, analysis revision 5. Correct completion within five minutes: absent 5/6, full 2/6, short 4/6. Seventeen artifacts passed. Token totals remain unknown; the original 11-slot cohort snapshot is preserved.
Read the pilot and next steps · Download revision 2 · Original 11-slot snapshot
AI-DLC: adding a structured workflow
October 16 · five repository pairs · 4/5 artifacts passed in both arms · inconclusive · publication 9. AI-DLC: 6.38× actor time, 13.68× tokens.
How study versions work
A study topic can have several separately registered cohorts. Each cohort freezes its protocol: task and check bytes, instruction arms, model, actor harness, permissions, delegation, budget and stopping rule.
- Protocol and cohort: a changed execution design or new budget creates a separately registered cohort. Results stay separate unless pooling was declared before outcomes.
- Analysis revision: a documented interpretation or evidence reconstruction using retained attempts. Keep original receipts, diagnostic evidence and the reason for any changed result.
- Publication revision: an immutable HTML/JSON snapshot with a date and change history. The stable study URL points to the latest reviewed publication; revision permalinks remain inspectable.
Schema labels such as “v1” describe file formats, not scientific design versions. “Completed” describes execution, not effectiveness. A workflow decision is distinct from a study verdict.
Model and harness coverage
The two TDD skill cohorts used requested gpt-6.1-sol with Codex 0.161 native execution. The separate AI-DLC cohort requested the same model through Codex 0.162.1. Smaller models and alternative actor harnesses have not been tested in these skill comparisons. Harbor supplied repository task transport and Linux verification; it was not a second actor harness.
A separate developer routing-scenario investigation exercised Claude Sonnet 4.6 and Codex, using different instructions and a different task. It is not cross-model validation of the skill findings. Read the benchmark-surface review.