Executed October 16, 2026 · Published October 17 · Analysis 1 · Publication 1
We added AI-DLC. Both workflows passed four of five tasks.
With AI-DLC, the five attempts used 6.38 times the actor time and 13.68 times the reported tokens of our existing workflow.
We compared our frozen coding setup with the same setup plus AWS Labs’ AI-DLC 2.11.0 Classic and an unattended decision adapter. The model and common tools stayed the same. This measures adding that package to our stack, not a bare default harness.
Four repositories passed in both arms. Both NodeBB submissions failed patch replay before acceptance tests. These counts do not establish equivalent quality or a general lack of benefit. Human collaboration and longer-term delivery were outside this comparison.
Same recorded success count, more measured effort
| Arm | Successful artifacts | Actor minutes | Whole-tree tokens | Unknown usage |
|---|---|---|---|---|
| Existing workflow | 4/5 | 49.98 | 22,023,383 | 0/5 |
| With AI-DLC | 4/5 | 318.69 | 301,314,459 | 0/5 |
Ratios use summed totals across all five assigned attempts per arm, including failures. Whole-tree tokens include descendants, cached input and repeated context; they are not billed dollars. Actor time excludes setup and grading. No experimental actor or calendar deadline was imposed.
The five repository pairs
| Repository | Recorded outcome | Existing workflow, min | With AI-DLC, min |
|---|---|---|---|
| NodeBB/NodeBB | Both replay failures | 6.3 | 74.4 |
| element-hq/element-web | Both passed | 6.4 | 81.1 |
| internetarchive/openlibrary | Both passed | 7.1 | 51.3 |
| navidrome/navidrome | Both passed | 24.5 | 68.4 |
| qutebrowser/qutebrowser | Both passed | 5.7 | 43.4 |
One attempt per arm on each repository. The task pool came from pinned SWE-bench Pro V2 repository changes and was reused from our earlier capped phase. Fresh isolated workspaces were used, but these are not held-out confirmation tasks. The data file lists exact upstream task IDs, links and input hashes.
NodeBB needs a qualification: both patches encountered appendonly.aof: already exists in working directory during pristine replay. We retain the recorded failures. Acceptance tests were not reached, so this is not evidence of two incorrect implementations. We did not repair the submissions or rerun the actors to improve the score.
What AI-DLC was meant to add
AI-DLC connects requirements, decisions, implementation and review through a persistent workflow. Its human-supervised delivery proposition is broader than our executable repository-patch task. See the pinned release and AWS’s explanation.
We used Classic with unattended decision adapter v1. The adapter is part of the treatment and is not a human reviewer. This result must not be described as testing unmodified human-supervised AI-DLC, every workflow profile, or every coding harness.
How we checked the work
Matched setupRequested gpt-6.1-sol, high reasoning, Codex CLI 0.162.1 and the same frozen instructions, skills, tools and permissions. Harbor 0.22.0 provided transport and verification. Both arms could delegate with the same limits.
Independent replayEach saved patch was applied to a fresh pinned task image, then checked using the benchmark’s independent acceptance tests. Actor-written test results did not determine the outcome. A non-applying patch failed at replay.
Workflow deliveryWe checked treatment exposure and reconciled root and descendant usage. Activation, artifact success and resource completeness were recorded separately. Closure and authentication removal were verified.
Fixed scheduleTen attempts across five adjacent pairs, baseline first in three families and AI-DLC first in two. No paid actor retries or outcome-driven task replacements. Literal null actor/calendar limits.
Tasks were classified as high complexity with our structural rubric; upstream metadata labels them medium. We did not compare moderate and high complexity. The selected changes should not be presented as a validated sample of long enterprise projects.
Every assigned attempt
| Order / repository | Arm | Artifact | Actor min | Tokens | Children |
|---|---|---|---|---|---|
| 01 · NodeBB/NodeBB | baseline | Replay failed | 6.28 | 4,339,270 | 3 |
| 02 · NodeBB/NodeBB | aidlc | Replay failed | 74.43 | 61,007,321 | 4 |
| 03 · element-hq/element-web | aidlc | Passed | 81.13 | 81,008,982 | 4 |
| 04 · element-hq/element-web | baseline | Passed | 6.43 | 3,540,894 | 3 |
| 05 · internetarchive/openlibrary | baseline | Passed | 7.12 | 5,645,575 | 3 |
| 06 · internetarchive/openlibrary | aidlc | Passed | 51.31 | 46,846,726 | 4 |
| 07 · navidrome/navidrome | aidlc | Passed | 68.43 | 60,476,556 | 4 |
| 08 · navidrome/navidrome | baseline | Passed | 24.46 | 4,513,729 | 3 |
| 09 · qutebrowser/qutebrowser | baseline | Passed | 5.69 | 3,983,915 | 3 |
| 10 · qutebrowser/qutebrowser | aidlc | Passed | 43.38 | 51,974,874 | 5 |
Recovery and earlier phases stay visible
Element Web’s AI-DLC actor completed naturally and saved its patch and usage before the worker was interrupted during independent grading. We retained the interrupted grader, verified the exact saved patch and re-ran only model-free verification. It passed. No actor was replayed. Its reservation-through-grading time includes the interruption and recovery delay, separately from native actor time.
The initial uncapped Flipt pair is separate: both passed; AI-DLC used 71.1 actor minutes and 61,064,843 tokens, compared with 16.6 minutes and 4,033,682 tokens for the existing workflow. Those observations are included separately in the data file and excluded from the five-family totals.
Before these uncapped phases, a 50-minute actor cutoff governed an earlier pilot. That phase was retired when the user removed the cutoff. Its outcomes remain preserved; they are not relabeled as uncapped observations or pooled here. This follow-up tests a changed timing policy, not independent replication on an untouched sample.
What this result supports
On these tasks, we observed substantially higher time and token use from the added workflow, with no observed correctness advantage under the supplied checks. The practical choice for this setup is to keep AI-DLC optional while investigating where its additional process earns its effort.
This is not a general claim that AI-DLC cannot help. A study of ambiguous requirements, real human review, handoffs and subsequent maintenance could measure benefits our task did not. A future confirmation also needs more independent families, repetitions and a declared practical decision margin.
- Five repository families with one attempt per arm; descriptive exploratory comparison, formal verdict inconclusive.
- The baseline retains the same existing instructions, skills and tools; it is not a bare default harness.
- Treatment is AI-DLC 2.11.0 Classic plus unattended decision adapter v1, not the unmodified human-supervised workflow.
- Both NodeBB submissions failed pristine patch replay before independent acceptance tests; these are not demonstrated behavioral implementation failures.
- This uncapped diagnostic reuses the task pool from the earlier capped phase; it is not a held-out confirmatory cohort. Fresh isolated workspaces were used.
- High complexity was assigned using the study structural rubric; upstream task metadata is medium. No moderate-complexity comparison was run.
- Reported tokens include cached input and repeated context; they are usage, not billed dollars.
- Native actor time excludes setup and grading. Element Web AI-DLC end-to-end time includes an interruption and model-free regrading.
- Only one requested model/harness combination was tested; no claim about Claude Code, other models or lighter AI-DLC profiles.
- Independent checks establish the supplied acceptance conditions, not absence of every possible defect or long-term maintainability.
Evidence and publication history
Completion: October 16 at 23:50 UTC. Five independent repository families, ten observed assignments, none unattempted. Formal verdict: inconclusive. No automatic workflow change.
Publication 1 is the first sanitized report of the completed uncapped expansion. Previous TDD cohorts and capped-phase records stay separate. Raw transcripts and patch files remain private; the download contains measurements, upstream task identities and exact evidence/artifact hashes.
Frozen registration SHA-256: b1ca5ce8264014a2b9d5b0d87d0aa8336fabde8d6483a09c3feb4e827e36a477
Completed execution SHA-256: 88265a0c1b653dc437b1c20c1dceaaeedfe42061b3f076f6f88775f1474b4be7
Runtime SHA-256: 8f9722afb16180758b67ff2f1fdf6591b77b549f953cac032d7d50c9abf8cb23
Publication 1 · Data revision 1 · Index and publication hashes