songjiun10-collab/hncs/verifiability-first

Verifiability First

Pin the success criterion first

If you set the bar after seeing the result, anything can be made to look like success. So before starting, decide "what has to be true for this to count as done."

• "Make it work" → what counts as working? Turn it into something verifiable.
• "Add validation" → write a test for invalid input first, then make it pass.
• "Fix the bug" → write a failing reproduction first, then make it pass.
• "Is A better than B" → decide what threshold counts as a win before comparing (X% or better, N out of M cases).
• When measurement or an experiment is involved: if the difference could plausibly be explained by chance, "inconclusive" is the correct answer, no matter how good the mean looks. Admitting you can't conclude is itself a result.
• Concrete case (Hncs): hybrid_engine's summarize() is the criterion pinned in advance, not chosen after looking at results — every comparison runs a paired t-test and a sign test and a bootstrap 95% CI (20000 draws, fixed seed) and a drop-one sensitivity check, and a CI straddling zero is defined as inconclusive regardless of the mean. It also requires a null result to clear a positive control before being trusted — two near-misses in that codebase (an X-Trans demosaic path collapse, an OMP_NUM_THREADS leak that silently blacked out 75% of every frame at exit code 0) would have been reported as "the knob does nothing" if a positive control hadn't caught them. Three past "decisive wins" in the same repo later turned out to be noise or an environment leak — the rule exists because a good-looking mean has already fooled that project multiple times.
• Principal angle: for any threshold or benchmark other teams will later cite as ground truth, the criterion itself becomes precedent — get it agreed before the numbers exist, not after.
• Distinguished/Fellow angle: if this benchmark could end up cited in a conference talk, a blog post competitors read, or an industry comparison, the criterion has to survive scrutiny from people with no memory of why it was chosen, years from now.
• Executive angle (CTO/VP-Eng): if the criterion will justify a budget, headcount, or vendor decision to the board, pin it before the numbers exist — otherwise the spend gets approved on a threshold picked to match the result.

A strong criterion lets you run to completion alone; "make it good" needs constant check-ins at every step.

Design for testability

Thinking about how you'll verify something up front naturally improves the design. Code that's hard to test is usually code with tangled responsibilities.

• How will you confirm this logic runs correctly?
• Does verifying it really require a DB, network, time, or randomness? Can that piece be isolated?
• Think of one failing case first — that's effectively your first test.
• Multi-step work becomes a plan with a verification step attached to each item.