What happened
Most of our server tests run against an in-memory Postgres. A few
can't: they need a real database, so they run from their own script
against a throwaway one. To keep them out of the ordinary run, they
skip themselves unless the database's name ends in _qa_e2e.
One of those suites checks that a workflow's steps really write
rows: a "create a ticket" step should leave a real ticket behind. It
had fourteen tests. One was failing with
relation "support_settings" does not exist.
The product was fine. Production has that table. The test's own database didn't: the script built the tickets table from a hand-written copy with eight columns, and the support inbox had since grown the real table and added a settings table beside it. The copy never followed.
Why nobody saw it
The ordinary run skipped the suite, as designed. The script that runs it was run by nobody: not by CI, not by a hook, not by a person. It only came to light when we ran every script by hand during a pass on the workflow engine.
A skipped test and a passing test look the same in a summary line. That is the whole problem.
What we changed
- The test database is built from the product's real tables, not a copy. The script applies the same migrations production runs.
- A guard in the ordinary suite, which does run on every change, compares the script's tables with the module's, column for column, and checks that every migration of that module is in the script's list, in order. If the harness falls behind again, an ordinary test goes red.
- A rule for ourselves: before merging a change to workflows, approvals or the ledger, run the real-database scripts by hand, and say so in the pull request.
The lesson
A suite the ordinary run skips is run by nothing, unless something in the ordinary run watches it. If you have tests behind an environment switch, add one cheap test in front of the switch that fails when the hidden ones would rot.
ADLR is at https://adlr.app: $5 per user a month, every module included, 30 days free with no card.