Adding a software factory workflow to codex: same pass count, more time and tokens.

Both workflows passed four of five repository tasks. With AI-DLC, the attempts used 6.38× the agent time and 13.68× the tokens of the existing workflow.

What I compared

After testing an adversarial TDD skill, I wanted to know whether a broader delivery workflow would earn its extra effort on my specific repository work. This is basically a software factory which was trending on Linkedin and I wanted to compare whether or not the increased overhead of the requirements gathering, decisions, implementation and review through the persistent workflow would actually improve results noticeably.

Each task ran once with this repositories existing instructions, skills and tools, and once with that same setup plus the AI-DLC Classic workflow profile. An unattended adapter answered approval prompts using a fixed policy and recorded the approvals as automated. Both variants used the same Codex model, harness, project instructions, skills and tools; one also used AI-DLC Classic.

The tasks were deliberately selected for bounded features with interacting components, error cases and independent acceptance checks. The pool covers API, UI, search indexing (Open Library's code for adding, updating and removing works, editions and authors in its Solr search system), database and parsing work.

Variants

Scroll across for all results. Select a heading to sort.

baseline = existing workflow; v1 = with AI-DLC. Totals include all five attempts per variant, including failures.

The existing workflow used about 50 minutes of agent time and 22.0 million tokens; AI-DLC used 319 minutes and 301.3 million tokens. Totals cover all five attempts per variant and the main and child Codex sessions, including cached input and tool output returned to the model. Agent time excludes setup and grading.

Cases

baseline: appendonly.aof already exists in working directory; acceptance tests were not reached; 6.28 min; 4,339,270 tokens
v1: appendonly.aof already exists in working directory; acceptance tests were not reached; 74.43 min; 61,007,321 tokens

baseline: passed; 6.43 min; 3,540,894 tokens
v1: passed; 81.13 min; 81,008,982 tokens

baseline: passed; 7.12 min; 5,645,575 tokens
v1: passed; 51.31 min; 46,846,726 tokens

baseline: passed; 24.46 min; 4,513,729 tokens
v1: passed; 68.43 min; 60,476,556 tokens

baseline: passed; 5.69 min; 3,983,915 tokens
v1: passed; 43.38 min; 51,974,874 tokens

Scroll across for all results. Select a heading to sort.

1 = artifact passed; 0 = patch replay failed. Open a prompt for time, tokens and task sources.

Saved patches were replayed on fresh task repositories using the task-specific SWE-bench Pro V2 acceptance tests. Open Library used openlibrary/tests/solr/test_update_work.py. Both NodeBB patches failed to apply before testing; the other four repositories passed in both variants.

What I take from this

The added process used more time and tokens without an observed advantage on these checks. As I'm realizing more and more, the tasks these tools are being asked to run are potentially more complex than the ones I'm asking it to do but my practical lesson: before standardizing a workflow (even one that's experiencing an explosion of popularity), compare it with the setup you already use on a few real tasks. Decide what improvement would justify the extra effort. You can try this on Rulemetric. I will also be adding followup tests to check different lower powered models and other harnesses, for a more thorough study, you'll be able to see those results below:

All the caveats, plus some explanation about the plan

This is an exploratory pilot: five repository pairs, one run per variant, and a formal verdict of inconclusive. The equal pass counts do not establish equivalent quality. Human review, handoffs and long-term maintenance could offer benefits this unattended test did not measure.

Benchmark sources

This study used SWE-bench Pro V2 · Scale AI. Other candidates are Terminal-Bench 2.1, DeepSWE · Datacurve, GitHub's internal CheckpointBench, and RuleMetric's own fixtures.