Authorization pilot · October 26–27, 2026 · Publication 4
Can we improve the performance of a popular, official open source skill?
I'm continuing my series of tests to see how skills affect model performance, in this case an official open source skill Supabase Postgres Best Practices at the exact version used in this study.
What if we take this a step further and improve the "performance" of the skill, while also testing the skill with harder tasks? What would that look like?
First the baseline
We took the initial version of the skill, and ran a set of sql permission migrations spanning two separate changes. This was over 3 separate tasks.
Each task involved two separate sql migrations, spanning two separate changes.
All three tasks centered around changing the permissions of records, either at the database level via row level permissions or adding a flag. They all contained real data, and real migrations and as it turns out, it was way too easy for these model pairings.
The same success rate with increased token cost
| Model | Successful tasks in each condition | Tokens without → with skill | Token change | Minutes without → with skill | Time change |
|---|---|---|---|---|---|
gpt-6.1-sol |
4/4 → 4/4 | 947,584 → 1,130,587 | +19.3% | 6.45 → 7.37 | +14.2% |
gpt-6-luna |
6/6 → 6/6 | 1,495,280 → 1,764,272 | +18.0% | 5.51 → 6.79 | +23.1% |
claude-sonnet-4-6 |
3/3 → 3/3 | 792,351 → 1,234,167 | +55.8% | 10.27 → 16.13 | +57.0% |
claude-haiku-5-5 |
6/6 → 6/6 | 2,832,850 → 2,579,138 | -9.0% | 14.79 → 14.39 | -2.7% |
Most of the iterations took more time with the skill vs. without it, with the notable exception of Haiku which decreased both tokens and time.
RuleMetric does a decent job at estimating prices, but these are still estimates.
A harder baseline: activity feeds and job queues
Since the original tests were clearly not difficult enough for all the combinations of models and agents, we tried other, more difficult tasks that we could use to improve the core skills.
This time we submitted sql against a real database, updated permissions and attempted to optimize db operations.
We also increased the number of harnesses (including copilot cli and pi in the mix). As a baseline for the skill, and we saw similar results with a couple of task failures and a more dramatic increase in the token and time count.
| Tool / model | Passed: no skill → original | Tokens: no skill → original | Token change | Time change |
|---|---|---|---|---|
| Claude Code / Sonnet | 3/4 → 4/4 | 4,532,925 → 5,336,206 | +17.7% | +7.6% |
| Codex / Sol | 4/4 → 4/4 | 1,183,592 → 1,469,407† | Unknown† | +11.8% |
| Claude Code / Haiku | 4/4 → 3/4 | 2,749,120 → 3,137,085 | +14.1% | +6.2% |
| Codex / Luna | 4/4 → 4/4 | 1,092,718 → 1,921,418 | +75.8% | +27.0% |
| Pi / Sol | 4/4 → 4/4 | 1,110,527 → 1,397,386 | +25.8% | +10.4% |
| Pi / Luna | 3/4 → 4/4 | 997,391 → 1,307,504 | +31.1% | +21.7% |
| Copilot / Luna | 4/4 → 4/4 | 1,231,188 → 1,787,626 | +45.2% | +21.3% |
| Copilot / Haiku | 4/4 → 4/4 | 1,708,060 → 1,455,282 | −14.8% | −18.7% |
Because there were failures, some of these pass counts skew the token counts since this reduced the amount of tokens rendered.
Revision 1: check the SQL that will actually be delivered
Now that we had failures, we had traces and issues that we could use to hopefully improve the skill.
We added a short section to the skill asking to doublecheck wether or not the proper migrations ran (it's more detailed than that, but that was the gist).
We did three comparisons, one with no skill, one with the original and one with the modified skill.
| Tool / model | No skill passed | Original passed | Revision 1 passed | Token change vs original | Time change vs original |
|---|---|---|---|---|---|
| Codex / Luna | 4/4 | 3/4 | 4/4 | -3.4% | -2.6% |
| Claude / Haiku | 4/4 | 2/4 | 4/4 | +11.9% | -3.4% |
| Copilot / Haiku | 4/4 | 4/4 | 4/4 | -9.1% | -8.6% |
| Pi / Sol | 4/4 | 4/4 | 4/4 | -0.1% | +7.8% |
There were notable improvements in tokens used, but results were again skewed by the failure/pass rates causing less tokens to be counted, or more. You also see the outcomes of non-deterministic systems: on continued runs, tasks that previously passed then failed. Most of the pairings saw reductions in tokens and time, but you'd ideally do more iterations with this same prompt to get that number to a larger amount (+15%).
Notes and final thoughts
This study shows the difficulty in reviewing and updating skills in non-deterministic systems, but that you can update/optimize them in order to reflect your own unique workflows. We'd need to run this at a larger scale with more iterations on harnesses and model combinations. It's interesting to see skills having greater negative and positive effects on lower tier models.
Notes on what we used
Exact SKILL.md SHA-256 hashes for this revision-one comparison:
Original:
ad65e776f45e761bafc609d3f506c77c1032225aa65e0c4c02f241b8071e5855
Revision 1:
36614d61501b5d7a124f718dd1e8f6d6cb296eb41c515083dc91b655df77d6f6
The original pilot's downloadable evidence describes that pilot only; it should not be read as the raw dataset for the later comparisons above. The later attempt records, measurement reconciliation and failed setup history remain preserved separately.
Published versions of this page
Publication numbers version the article. “Revision 1” above names a tested change to the skill; it is not website publication 1. The main study URL points to the latest published article, while these links retain earlier text:
- Publication 1 — original pilot report and checker history.
- Publication 2 — shortened pilot write-up.
- Publication 3 — reframed around improving the official skill, published November 3, 2026.
What we used
The original pilot used Codex CLI 0.164.0 and Claude Code 2.1.310. Codex model names are requested identities; Claude identities were observed in the responses. The follow-up comparisons above include Codex CLI, Claude Code, GitHub Copilot CLI and Pi. Results apply to their recorded task/model/harness/version scopes; unavailable combinations and failed setup attempts remain separate from completed comparisons.
Supabase skill used in this study · Archived original report with the full method and results