mikecodeur/killer-saas/testing-doctrine

Testing doctrine

The rules here decide what gets written and what gets run. They live in a skill rather than in AGENTS.md because only two agents act on them — the one that writes the tests and the one that judges them — and every other context was paying to load them.

Budget: `Test budget` from `AGENTS.local.md` (25 by default), and a story that needs more says why in its plan.

Measured in production use: the suite grows by dozens of tests per story, and most of those stories still ship their central invariant with no net at all — every one found by mutation during review, none by the volume. The number does not measure the net. It buys false confidence, and it is slow: a suite that takes half an hour to run is a suite nobody runs.

Where the tests go

End-to-end, specifically

An end-to-end run costs a cold server, a migration, a seed and a browser, and the suite is replayed by every story after this one. Measured on a real project: 86 specs, 216 tests, seven minutes a run, and most of the reds were specs repaired for a wording change rather than a broken feature.

• One spec per story, one scenario in it — none when no unit test is structurally blind

to what the story does.

• It asserts an effect: a URL reached, a row persisted and read back, a real HTTP status,

a refusal enforced by the server. A label, a heading or a link merely being visible is not an end-to-end assertion — if the screen must be looked at, look at it in a browser and record it.

• No matrix. Locales × themes × viewports, every route in a loop, every role: each one

belongs to the layer that owns it (policy test, i18n check, visual check), or nowhere. Written inside one test(), it still counts as what it replays.

• Select by role, or by a `data-testid` where the visible text is copy that can change.

A wording change must not break a spec.

• A story does not edit another story's spec, unless its own change legitimately breaks

it — and then the verification record says which, and why.

Four cuts, each measured in real use

1. The matrix once, in the policy test. Replayed per command, it takes a single service test file past fifty tests.
2. No enum exhaustiveness. Testing every ordered pair of a transition table proves nothing the legal transitions plus one representative refusal do not already say.
3. Never the same rule at two layers. Pick the layer where the rule lives.
4. No adapter re-asserting a 403.

No red-first ceremony, and no invariant mutations in implementation. Do not write a failing test to watch it fail, and do not neutralize a guard to confirm a test goes red. Write the code as whole blocks, write the tests that belong to it, run them. Measured: repeated mutations across a story and its fix pass, each costing a full suite run plus a restore plus another run, and zero findings — the author who just wrote the test already knows it passes. It verifies the tests, not the code.

The same technique stays in review, and there it earns its place. Run in fresh context on someone else's net, it found a critical, six majors, and — repeatedly — an invariant with no test at all. The safety moves downstream to where it works; it is not dropped.

The failure mode this pipeline keeps producing: a test that names an invariant without exercising it. Most stories ship one. Six shapes seen repeatedly, worth citing verbatim in a prompt because they are hard to spot by reading:

1. a hand-written query in the test instead of a call to the code under test — it tests the database, not the repository;
2. a payload the real interface never produces;
3. a catch that swallows the failure, so the assertion passes when nothing throws;
4. an end-to-end test that stays green while rendering zero rows;
5. a fixture whose identifier lets a downstream guard answer for the guard under test — invisible when reading assertions;
6. a mock double that replays the clause instead of evaluating it.

The criterion that replaces the count: a test that stays green when the rule it names is deleted is worse than no test — it hides the hole it claims to cover.

When to run what — each thing once, and once only

A deterministic command re-run on the same code returns the same answer. Running the suite in Execute, again in Review, again per mutation and again at Ship buys nothing and costs four times the wall clock. Each check therefore has exactly one stage, and the stage is a setting in AGENTS.local.md.

The reviewer's mutations run on the invariant's own test files, not on the suite. A neutralized guard turns red in the one or two files that name it; replaying the whole project to learn that is the single most expensive habit this pipeline ever had.