Do Supabase skills help agents keep access rules correct?

Within-model outcomes

A strict pass requires both the initial migration and its follow-up to pass every check, successful native completion, and an unchanged first migration. Each row compares matched family/repeat pairs under the same model, harness and execution policy.

No skill = identical tasks, schema, tools and official documentation without the optional skill. Corrected = the post-hoc checker replay. Repeats, stages and assertions do not create additional independent task families. Matching pass counts do not establish equivalence.

One Sonnet document-sharing pair spans policies: its baseline began before the cap was removed and finished afterward; its skill arm ran uncapped. Both artifacts appear in the family rows and download, but this pair is excluded from the matched comparison table.

Effort for the same matched pairs

For gpt-6-luna, the skill arm used 1.18× the no-skill token total. For claude-haiku-5-5, the skill arm used 0.91× the no-skill token total.

These totals include all sessions in each displayed arm, including unsuccessful results. Tokens include cached input. Elapsed time includes native transport setup and tool use; it is not just model latency. Other local work and verification replays overlapped some coding sessions. These are observed totals, not a causal estimate of savings.

Skill / no skill is the token-total ratio for that row only. No pooling across model, harness or policy. Native subscription usage is not incremental API spending.

What the agents had to build

All three are authored synthetic fixtures with five starting records, not production repositories. The checker exercises allowed and forbidden operations: cross-tenant reads/writes, forged owners, identity changes, viewer writes, share revocation, role escalation, TRUNCATE grants and exact preservation of starting data.

How the comparison worked

Each model has two repetitions per arm per family. Native CLI sessions use existing subscriptions, isolated homes, disabled capture and delegation, and an owned-container tool. The actor database has no external network. The skill is copied only into treatment workspaces; it is never globally installed.

The tool trace returned the skill header in 47 of 48 treatment sessions. Missing read evidence is listed in the download. This measures assignment to supplied guidance, not guaranteed consumption or compliance; no session was rerun for failing to read it.

Models: requested gpt-6.1-sol and gpt-6-luna through Codex CLI 0.164.0; observed claude-sonnet-4-6 and claude-haiku-5-5 through Claude Code 2.1.310. Codex JSONL does not attest the served model. The Codex catalog defaults differ (Sol low, Luna medium); Claude defaults were retained. The Haiku and Sonnet pins span generations. This is not a controlled comparison of model tiers.

The checker correction

The original negative checker accepted permission/RLS message strings or a successful zero-row operation. A legitimate immutability trigger could raise a different message and be scored as failure even though the update was refused. PostgreSQL supports custom exceptions and explicit SQLSTATE codes; see its error-reporting documentation.

The corrected replay adds one narrow allowance: for the two owner/org mutation probes, a PL/pgSQL RAISE with code P0001, 42501 or 23514 counts as refusal. All allowed-operation tests still have to pass. Syntax errors, division by zero, an unrelated RAISE code and triggers that reject every update still fail.

57 correction controls passed: the original reference/seeded-defect controls plus valid conditional triggers with all three codes and broken-trigger controls. The first narrower correction recognized only P0001; its retained replay exposed the other two codes. That refinement is part of the post-hoc history. The corrected checker, controls, exact hashes and both score sets are included in the download.

The frozen-checker verification replay covers 97 stage results, including the historical partial stage and its linked continuation. Exact mismatches: 0. This is artifact replay, not repeated model sampling.

Every family and stage

These rows use the final linked trajectory record; the download also retains the interrupted original record. “Fail” means at least one check failed, not that every operation was wrong.

Execution history and full accounting

The first run stopped after 17 coding sessions at an agent-imposed token threshold. The owner removed that cap and the similar execution ceilings. The linked continuation ran only previously unexecuted stages; completed passes and failures were never rerun. The smaller-model cohort used the uncapped policy from the outset.

Total unique coding sessions: 96; known coding tokens: 17,338,869; unknown coding counters: 0. Setup smoke checks, including failed earlier preparations: 16 sessions and 331,639 known tokens (0 unknown counters). The inherited first stage is counted once. Supervisor tokens are unknown; model-free checker and publication work are separate.

No total-token, turn, command-count, actor-output, actor-container resource or execution-clock ceiling remained in the uncapped profile. Existing trusted-verifier query limits and isolation stayed unchanged. Real subscription and hardware limits still apply.

Replay the evidence

The reusable kit contains the exact fixture, both checkers, task prompts and source hashes. The evidence contains submitted SQL and per-stage checks. Python 3.11+ and Docker are sufficient for artifact replay; no account, model subscription or database credential is required. Inspect the downloaded code before running it.

curl -fsSLo replay.py https://rulemetric.com/studies/supabase-rls-2026-10-26-replay-r1.txt
curl -fsSLo kit.json https://rulemetric.com/studies/supabase-rls-2026-10-26-kit-r1.json
curl -fsSLo evidence.json https://rulemetric.com/studies/supabase-rls-2026-10-26-r1.json
python3 replay.py --kit kit.json --controls
python3 replay.py --kit kit.json --evidence evidence.json   --checker corrected --case lower-models-01/public-catalog-2-codex-baseline

The kit replays synthetic artifacts in fresh, network-isolated Postgres containers. It does not reproduce stochastic native model calls. The public JSON is a sanitized projection. Original protocol bytes, native transcripts and capture-isolation receipts remain private; no credentials or raw session identifiers are published. The final read-only serving-database audit checked 112 native session IDs and found 0 normal-capture matches.

Provenance and limits

Candidate: Supabase Postgres best practices at the exact source commit. Individual skill files, documentation, prompts, source, CLI binaries and database image are hash-bound in the evidence. RuleMetric's study runner retained the comparisons and replay records; this was not a natural captured-session adoption loop.

• Original capped phase stopped after 17 sessions; immutable partial record retained. Uncapped continuation executes only unexposed stages.
• One Sonnet baseline trajectory crosses execution policies; its counterpart is excluded from matched same-policy comparisons.
• Custom-trigger denial correction was designed after inspecting failures. Both frozen and post-hoc scores are retained.
• Only three authored families with small fixture tables. No production workload, performance benchmark, hosted Auth/PostgREST, automatic skill routing, or broad security certification.
• Codex identity requested-only; Claude observed model checked. Native defaults and generations differ; within-model comparisons only.
• Cached input included in tokens. Subscriptions used; tokens and CLI list-price estimates are not incremental dollars. Supervisor tokens unknown.
• Assignment to supplied skill does not guarantee consumption or compliance; observed skill reads and missing read evidence are reported separately.
• Local work and model-free verification replays overlapped some coding sessions; elapsed totals are descriptive rather than isolated latency measurements.
• No optional skill was installed, removed, or adopted based on this pilot.

The useful next step is fresh, independently selected tasks under a corrected preregistered checker. This pilot is evidence about these supplied instructions on these fixtures, not a reason to disable unmeasured skills.