Source review · October 12, 2026 · Publication revision 1
How our agentic benchmarks show their evidence
Public studies, saved comparisons and developer benchmarks share a need for frozen identities, complete attempt accounting and visible limits. Their displays currently differ.
Authenticated live browser behavior was not newly certified by this review. Private benchmark records are not made public by listing their display conventions here.
Current surfaces
| Surface | Current evidence display | Qualification or gap |
|---|---|---|
| Public skill studies | Static HTML, sanitized per-cell JSON and immutable revision links. | October 10 completed; October 12 incomplete. Verdict and execution status remain separate. |
| Saved comparisons / Evals | Canonical experiment inventory; observed/planned counts, named metrics and actions. Detail shows frozen protocol hashes, arm bytes and fixture/check versions. | A common topic/cohort and publication-history label across authenticated views remains future work. |
| Harbor / benchmark executions | Frozen benchmark artifacts and canonical experiment lifecycle; developer scripts also show reward/resource summaries. | Task transport, verifier and actor harness are separate. Reward and complete within-budget execution differ. |
| Target studies and live confirmations | Task/repetition, assigned/observed/missing counts, scope/model, resource uncertainty and application links. | Applications and confirmation are separate facts; publication/cohort history is not yet uniform. |
| Promptfoo routing scenarios | Scenario revision directories, engine/candidate-specific results and authoritative actor/verification receipts. | The Claude Sonnet 4.6 and Codex routing task is exposed exploration, not replication of the skill studies. |
| Complex manual artifact tasks | Separate saved comparison and developer receipt, with checks and native delivery witnesses. | Author-selected tasks share a source lineage; passing outputs do not establish general coding efficacy. |
| Instruction matching replay and outcome checks | Frozen dataset/plan/run hashes, reference provenance and review/report exports. | Distinguish retrospective/prospective samples, automated-reference agreement and executable task benefit. |
| Mechanism/security controls | Regression and calibration reports, CI output. | Control checks do not add independent task families or prove agent efficacy. |
Versioning for continued updates
Freeze each cohort’s protocol and execution budget. Use separate analysis revisions for retained-evidence corrections, and immutable publication revisions for changed pages or exports. Preserve prior snapshots and the reason for every changed result.
The public study index now implements publication history for the two published skill cohorts. A uniform authenticated display of topic/cohort, protocol identity, analysis revision, evidence cutoff and publication links still needs a separate application change.
Do not pool different candidates, models, harnesses, task populations or budgets after seeing outcomes. A format schema version alone is not a scientific protocol version. Failed, unknown and unattempted cells remain visible.
Coverage and further work
Both full-skill studies requested gpt-6.1-sol with Codex 0.161 native execution. Neither tested smaller models or compared alternative actor harnesses. Harbor supplied task transport and Linux verification in the repository pilot.
A separate routing-scenario comparison used Claude Sonnet 4.6 and Codex with different instructions on a different task. It does not validate the skill findings across models.
Next, review the stopped repository pilot using retained evidence only. Any new model or actor-harness experiment needs a separate frozen design and budget; broad confirmation also requires untouched representative families.
Repository pilot and next steps · Published studies and version history