Authorization pilot · October 26–27, 2026 · Publication 2

Does a popular, official Supabase skill improve model performance?

Across all variants of models and harnesses, wether the skill existed didn't change the success outcome of the different tasks. All tasks including the skill took longer to complete.

The same database tasks reached the same final success rate with and without the Supabase skill. In the matched uncapped comparisons, three of four model groups used more tokens and took longer with the skill. Haiku was the exception, using slightly fewer tokens and less time.

We tested Supabase Postgres Best Practices at the exact version used in this study. We didn't test any other skills besides this one in the skills repo

The tasks we asked the models to complete

Each task involved two separate sql migrations, spanning two separate changes.

All three tasks centered around changing the permissions of records, either at the database level via row level permissions or adding a flag. They all contained real data, and real migrations and as it turns out, it was a fairly easy task.

These were two-stage database changes. Each model first wrote a SQL migration, then received a new requirement in a fresh session. It had to add a second migration while leaving its first migration unchanged.

Let viewers read their own organization’s records

First, allow editors to access records belonging to their organization. Then extend read access to viewers in the same organization, while keeping writes restricted. The checks tested whether viewers could write, whether a user could access another organization’s records, and whether someone could forge or change a record’s owner or organization.

Revoke access to a shared document

First, let an owner share a document with another person as read-only. Then make sharing depend on an active flag and revoke access immediately when the share was disabled or deleted. The checks exercised both legitimate access and attempts to keep reading or writing after access should have ended.

Keep unpublished catalog entries private

First, allow public reads of a catalog. Then make unpublished drafts private while preserving editor access and publication changes. The checks tested public visibility, editor permissions, and separation between organizations.

All three were authored fixtures with five starting records. We checked the submitted SQL using real database roles, including allowed actions, forbidden actions, and preservation of the starting data.

The same success rate with different effort

Sol, Luna, and Sonnet used more tokens and time without improving the measured success rate. Haiku used 9.0% fewer tokens and took 2.7% less time. Tokens include cached input; elapsed time includes tool use and transport setup on a shared machine. These are observed differences in this pilot, not established speed or efficiency effects.

A correction matters to the result: our original checker rejected some valid ways of refusing forbidden database changes. We corrected those checks after inspecting failures and replayed the same submitted SQL. No agent was rerun and no submitted code was repaired. The 100% result comes from that post-hoc replay; the original scores remain archived.

What the differences cost

For example, Sonnet’s matched uncapped tasks cost an estimated $0.44 each without the skill and $0.65 with it, about $0.21 more for the same measured success rate. Luna’s increase was about $0.00054 per task. Haiku’s estimate fell by about $0.00136 per task.

RuleMetric does a decent job at estimating prices but these are still estimates.

What we used

The original pilot used Codex CLI 0.164.0 and Claude Code 2.1.310. Codex model names are requested identities; Claude identities were observed in the responses. The follow-up plans Codex CLI, Claude Code, GitHub Copilot CLI, and Pi, with unavailable model/tool combinations recorded separately (and will be added later on).

Supabase skill used in this study · Archived original report with the full method and results