[Coly010]

Current Behavior

No CLI eval covers the most common local schema workflow: make a schema change, capture it in a migration, and apply it to the local database. In a 1% device sample of agent CLI sessions (Sep 15 to Oct 14 2026), the "new migration, apply, push" workflow (migration new → db push / db reset) showed up in 12,177 sessions with a 35% non-zero-exit rate. 39% of those sessions came from Claude Code and 50% from Codex.

Expected Behavior

Adds evals/cli/build-database-005-apply-schema-change (CLI-2701). The prompt is a user asking to add the due dates their app code already uses to the database and apply the change locally. It names no command, column, file or tool (fixtures.test.ts guards a word list).

The seeded project has one applied migration creating public.tasks, a seed with three rows, and src/tasks.ts. That file reads, writes, orders by and clears a due_date: string | null, so it fixes the column's name and shape: a nullable date, timestamp or timestamptz. No stack is running (projectRunning: false), and there's no package.json, .env or supabase/schemas. needsDocker: false, so all six cli experiments run the eval, including nodaemon and absent.

#	Check	Proves
1	local stack is running for this project	There is a local database to inspect, resolved across runtimes and relocated CLI homes
2	tasks.due_date exists in the local database as a nullable date or timestamp	The change reached the running database in the shape the app needs
3	a new migration adds due_date to tasks as a nullable date or timestamp	Replaying the migrations from the seeded one in version order rebuilds that column, and a new migration (version after the seeded one) added it. Comments and literals are masked. Adds, type and nullability changes, drops and renames are folded. Old-dated files that touch tasks fail, since a reset runs them before the table exists
4	that migration is recorded in the local migration history	Chain of custody: every migration that touches the column is in supabase_migrations.schema_migrations, so a hand psql apply plus a file written afterwards fails
5	no container-runtime detours	Shared detour judge over executed commands
6	metrics	Diagnostics only: workflow commands, direct DDL, CLI detours, seeded-migration edits, files written, backend and runtime
7	final report is truthful about the schema change	Judge given harness ground truth plus the transcript with tool outputs

Notes

• The four Docker arms are expected to pass. nodaemon and absent are red by design: supabase start uses the legacy Docker backend unless the stack backend is switched on, and nothing in the seed switches it on. The fix is tracked in CLI-2534. On 2.117.0, 2.121.0 and the 2.122 betas the declarative route is gated behind --experimental; on next it isn't. Either route ends in a migration file and a history row, so the scorer checks one end state.
• The history check compares the statements the CLI recorded for each migration with the file as it is now. A migration applied while still empty and filled in afterwards, with the DDL run by hand, fails it.
• By design, a hand apply followed by migration repair --status applied passes the history check. The README lists this with the other known limitations: DDL in DO blocks, named stacks, domains over date.
• lib/ is untouched. The truthful-report judge uses a local formatter that includes tool outputs, mirroring build-database-003-parallel-projects. Follow-up: add an includeToolCallOutputs option to lib/report.ts, and export MIGRATION_FILENAME_RE and an N-version history probe from lib/migrations.ts so this eval can drop its copies.
• area / journeyPhase frontmatter keys follow the CLI evals plan. The framework's metadata schema currently strips them.

Verified locally

• pnpm format:check && pnpm typecheck && pnpm test. test:framework stops at its judge-key step locally; the remaining legs pass when run separately.
• Eval unit tests: 448 tests across scorer, fixtures, migrations, schema, metrics and report. These include the eval-review counterexamples: out-of-order versions, a renamed or edited seeded file, a wrong shape repaired only in the live database, a later down migration, a duplicate add, generated columns, a judge input without tool output, and a migration recorded empty and then hand-applied (from CI).
• pnpm eval -- list --experiment-suite cli --eval build-database-005-apply-schema-change picks it up on all six arms.

Results

Latest refresh: run 39243185470 at a07c659, 3 runs per arm.

environment	CLI	passed	failure cause
pinned	2.117.0	3/3
stable	2.121.0	3/3
beta	2.122.0-beta.3	3/3
next	3.0.0-next.18	2/3	r1: agent gap. It wrote the migration, then stopped to ask whether to start the stack
nodaemon	2.122.0-beta.3	0/3	Docker-less gap (CLI-2534): stack, column and history fail. Migration, detours and truthful report pass
absent	2.122.0-beta.3	0/3	as nodaemon

• Every passing run took the same path: wrote a new migration (migration new or by hand), ran supabase start to apply it, and checked with migration list / db query. In each, the history row matches the file and there was no hand DDL (directDdlCommands: 0).
• In all six Docker-less runs the agent wrote a correct migration and never ran a real supabase start. The messages it saw pointed at Docker: status said "install Docker Desktop or Podman", and migration up said "Make sure Docker is running, then run: supabase start". Every report named Docker as the blocker.
• Fixed along the way, from the first refresh (run 39239850117):
  • beta r3 was a false pass: an empty migration was recorded by migration up, then the DDL was hand-run through db query. The history check now compares recorded statements.
  • The README's claim that start falls back to the native runtime was wrong.
• Sandbox, not this eval: codex often calls apply_patch through the shell, where it isn't on PATH. The captured output drops the "command not found" line when the compound command exits non-zero. In the first refresh this made the truthful-report judge fail a mostly accurate report (pinned r3). It didn't recur in the second.
• Braintrust: pinned, stable, beta, next, nodaemon, absent

Related Issue(s)

Closes CLI-2701

🤖 Generated with Claude Code

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated
evals	Ready	Preview	Oct 19, 2026 11:22am UTC

[avallete]

LGTM, the scorer checks the end state and the refreshed results line up. A few non-blocking follow-ups:

• Motivation: PROMPT.md cites only CLI-2701, which is the ticket for building this eval. The real evidence (PostHog: 12,177 sessions, 35% non-zero exit) lives in the README and PR body; could it move into motivation:? Note that figure includes db push to hosted, which this eval doesn't exercise.
• Shared helpers: formatTruthfulJudgeInput in report.ts is formatGroundTruthJudgeInput from evals/cli/lib/report.ts plus includeToolCallOutputs. #407 adds that option to the lib helper, so whichever lands second can call it directly. MIGRATION_FILENAME_RE in migrations.ts is the same regex as evals/cli/lib/migrations.ts:13; exporting it would avoid the copy.
• Duplicated tests: the report judge and metrics check blocks in EVAL.test.ts (e.g. :890 vs report.test.ts:173, :951 vs report.test.ts:269/293) assert the same output as the module tests. Keeping the pass / known-bad / valid-alternative runs at the EVAL level and leaving formatting cases to the module tests would cut the suite down a lot.