thefrederiksen/devthrottle/checks-that-fail-open

Checks That Fail Open

A check FAILS OPEN when its pass condition is an ABSENCE, and FAILS CLOSED when its pass condition is a specific PRESENCE.

That is the whole law. Doing nothing at all satisfies an absence. Doing nothing produces the wrong value against a presence.

Written up 2026-08-07 after this fleet hit the same defect eight times in one day, in eight different media, and nobody recognised it as one thing until they were listed side by side. The ninth will be somewhere nobody predicted, which is why this is a skill and not an issue.

The root

Every instance is a measurement standing in for the thing it measures.

And the honest reason it goes unnoticed: the tool usually makes the weaker claim cheap and the stronger claim effortful. message send reported delivery in one command; the blocking ask (since removed) waited for the agent's own answer. Two careful sessions used only the first, all night, without noticing. Today message send answers only "queued", and the stronger claim is a reply you asked for with --reply-wanted and read from your inbox.

The eight that cost a day

1. A sweep that searched nothing. Run by absolute path from outside a repository, git grep errored, the return code was ignored, and it printed files searched: 0 / UNANNOTATED CLAIMS: 0 and exited 0 - certifying the whole repository while searching nothing.
2. An `awk` range that never matched. It printed no delete-shaped calls and looked exactly like proof - inside a writer's own verification of a fix to an overclaim.
3. `git grep` with a leading slash, on Windows. Git Bash rewrites a leading-slash argument to a Windows path before git sees it, so /docs/start arrives as C:/Program Files/Git/docs/start. No error, no warning, exit 0. A sweep run exactly as written in the plan reports a clean pass having searched for the wrong string. Guard with MSYS_NO_PATHCONV=1, drop the slash, bracket it as "[/]docs/start", or use ripgrep.
4. A guard that cried wolf on prose ABOUT the defect. A grep for a bad address also matched the comment discussing that address - so it fired on exactly the file it existed to guard, teaching whoever ran it to wave the hit away.
5. A range anchored on a heading. Rename the heading and the range yields nothing, the grep prints zero lines, zero lines contain no bad string - a pass, while the check ran over nothing.
6. A review. Three review seats died in one night; one had completed its review and recorded the verdict NOWHERE. "No defects were posted" is an absence, and is satisfied by a review that never ran.
7. A success test reading a truncated tail. Delivered to <long name> (1 session(s)). wraps; | tail -1 showed only session.; the test looking for Delivered failed, and four identical messages all landed while the check said none had.
8. A shell that ate the content. A gh pr comment --body containing a backticked filename had it command-substituted away. gh posted, returned a URL, and exited 0 - and the published sentence was missing the two words it existed for. A tool that returns a link is not a tool that returned your content. Use --body-file with a quoted heredoc.

The repair, which is always the same

Do not fix an absence-shaped check with more care. Restate it as a presence.

And the presence must not filter on the property the defect would fail. This qualifier is load-bearing, and it was learned by watching the law applied mechanically produce a new hole. Restating "the stylesheet is stamped" as the specific presence count(stamped URLs) == count(stamp placeholders in the template) is a presence, it is specific, and it CANNOT FAIL: add a file that was never stamped at all and both numbers are unchanged. The failing case sits outside both sides of the comparison. A presence filtered on the property under test is the same hole with a number in front of it - see the count-shaped section below.

The presence that works is a DERIVED enumeration with a per-item assertion: not "four URLs are stamped" but "here is every static URL found ON THE PAGE ITSELF, and each one ends with the stamp".

DERIVED is the whole of it, and it is what reconciles this with "an enumeration is an absence claim wearing a list's clothes" under the table. An enumeration you HAND-KEEP is exactly that - it is true only if exhaustive and it cannot prove its own exhaustiveness, and the hand-kept list in the count-shaped section below went stale exactly that way: it named three files while the template stamped four. An enumeration DERIVED from the artifact under test - every URL the rendered page actually contains, every input tag the template actually declares - is exhaustive by construction, because the thing being checked is the thing being enumerated. If you had to type the list, it will go stale. If the check reads the list off the artifact, it cannot.

An enumeration is an absence claim wearing a list's clothes - true only if exhaustive, and an enumeration cannot prove its own exhaustiveness. One page asserted five event handlers where there were six: the conclusion was right and the evidence was wrong, which is the worst combination, because it survives review.

Five rules that fall out of it

1. State all three arms. A reader assumes two outcomes. There are three: the expected result is a pass, a bad result is the defect, and an EMPTY result is a repair job on the instrument - never a clean run.
2. Quote the actual output, never the word "passed". A load-bearing count is only load-bearing if it is visible in the record.
3. Reconcile a sweep against its own expected inventory. Name what you expect it to find FIRST, run it, then compare. A count with nothing to compare against proves only that the tool ran - and a truncated sweep is indistinguishable from a clean file. One sweep covered 23 of 37 and reported clean; it was caught only because the missing items were known to exist.
4. A control proves the check RAN, not that it reached FAR ENOUGH. A line count is a control for TRUNCATION, not for SCOPE. An event, a callback, a hook, a registration - any fan-out is a REACH problem, and only enumerating the subscribers answers it.
5. Run it against a known-BAD input. A check only ever run against the state you hope passes has demonstrated nothing. Prove it FIRES as well as passes.

The subtler sibling: a claim that is RIGHT resting on evidence that is WRONG

Harder to catch, because the conclusion survives scrutiny precisely because it is correct - so nobody re-examines what it stands on.

Seen four times in one day, every one found only because a reviewer opened the cited range: a correct negative proved by an incomplete enumeration; a correctly narrowed claim citing a comment about a different class; a correct exoneration resting on a comment saying so; and a real experiment with a correct conclusion whose one wrong detail would have argued for the wrong practice while citing the experiment as proof.

A wrong conclusion gets argued with. A right conclusion with false evidence gets agreed with, cited, and inherited - and the falsehood travels inside it.

Agreeing with a conclusion is not a reason to skip its citation - it is the situation where skipping is most likely and most costly. And when you cannot find evidence for something you believe is true, say less rather than reaching for the nearest plausible citation.

Applying it to review, which is where it costs most

• State a PASS as explicitly as a defect, naming what you checked AND what you could not reach. A pass with no stated scope cannot be judged by anyone who was not there - and everybody who was there is exactly the population that disappears. Report it wherever the review was asked for; if you have been asked to publish it somewhere, publish the same scope there.
• Name the artifact, not just "durably". Where a verdict is recorded decides whether it survives, and the tiers are not equal: a comment on the change beats a pushed file beats a committed file beats a local one beats a message that scrolls away. "Durably" with no named target lets someone pick the weakest rung and believe they complied. Do not write to anywhere you were not asked to write - a review that posts itself somewhere unexpected is its own defect.
• A verdict on file that covers a commit which is no longer the head is not a verdict on the head. A verification has a timestamp.
• A comment describing a fix is itself a claim, and goes stale like any other text. A durability rule manufactures this surface: telling a fleet to write everything down produces comment blocks, inventory rows and defect narratives, all of which read as commentary and none of which is exempt. The rule ships with the leg-hunt attached.

The COUNT-SHAPED check, and three siblings (2026-08-20)

Four more, in one evening, between two sessions. The first three are one shape and it is not covered above: it is not an absence, it is a comparison of two numbers where BOTH SIDES FILTER ON THE PROPERTY THE DEFECT WOULD FAIL, so the failing case sits outside both sides and is invisible to the comparison. The check is a presence, it is specific, and it still cannot fail.

• A hand-kept list naming three stamped static files while the template stamped FOUR. The fourth

arrived with a later feature and nobody updated the list; the test passed on the three it knew.

• assert page.count(stamp) == template.count(stamp_placeholder). Add a fifth static file with

NO stamp and both numbers stay four - counts agree, test passes, page ships an unstamped file. Measured ALIVE by mutation before the repair.

• assert page.count("data-default") >= 2 on a slider-reset control. It counts only the sliders

that HAVE a default, so a slider with none is invisible and never gets reset. Measured ALIVE.

One sentence: a test checking a NUMBER ABOUT the thing instead of the thing. The repair is always to enumerate and assert the property of EACH item - every static URL carries the stamp; every range input carries a default.

How the third was found, which is the transferable part. Not by re-reading, and not by grepping for count(. By searching the suite for the DEFECT'S SHAPE: which assertion compares two numbers where both sides filter on the property the defect would fail. Searching by the TOOL (ast.walk, rglob, count() finds duplicates of a mechanism. Searching by the SHAPE finds the same defect in code that shares no tool with it - a stamp count and a slider count have nothing in common but the shape. When you fix one instance, search for its shape before you close it.