Authorization pilot · October 26–27, 2026 · Publication 3
Can we improve the performance of an official skill?
I'm continuing my series of tests to see how skills affect model performance, in this case an official open source skill Supabase Postgres Best Practices at the exact version used in this study
What if we could improve the performance of the skill, what would that look like?
First the baseline
We took the initial version of the skill, and ran a set of sql permission migrations spanning two separate changes. This was over 3 separate tasks.
Each task involved two separate sql migrations, spanning two separate changes.
All three tasks centered around changing the permissions of records, either at the database level via row level permissions or adding a flag. They all contained real data, and real migrations and as it turns out, it was too easy
The same success rate with different effort
| Model | Successful tasks in each condition | Tokens without → with skill | Token change | Minutes without → with skill | Time change |
|---|---|---|---|---|---|
gpt-6.1-sol |
4/4 → 4/4 | 947,584 → 1,130,587 | +19.3% | 6.45 → 7.37 | +14.2% |
gpt-6-luna |
6/6 → 6/6 | 1,495,280 → 1,764,272 | +18.0% | 5.51 → 6.79 | +23.1% |
claude-sonnet-4-6 |
3/3 → 3/3 | 792,351 → 1,234,167 | +55.8% | 10.27 → 16.13 | +57.0% |
claude-haiku-5-5 |
6/6 → 6/6 | 2,832,850 → 2,579,138 | -9.0% | 14.79 → 14.39 | -2.7% |
Sol, Luna, and Sonnet used more tokens and time without improving the measured success rate. Haiku used 9.0% fewer tokens and took 2.7% less time. Tokens include cached input; elapsed time includes tool use and transport setup on a shared machine. These are observed differences in this pilot, not established speed or efficiency effects.
RuleMetric does a decent job at estimating prices but these are still estimates.
What we used
The original pilot used Codex CLI 0.164.0 and Claude Code 2.1.310. Codex model names are requested identities; Claude identities were observed in the responses. The follow-up plans Codex CLI, Claude Code, GitHub Copilot CLI, and Pi, with unavailable model/tool combinations recorded separately (and will be added later on).
Supabase skill used in this study · Archived original report with the full method and results