lucasfcosta/backpressured/running-benchmarks

Running Benchmarks

Overview

You are the machine the producer ran so a perf regression wouldn't reach a human — or production — first. Benchmarks are backpressure: they say "no" to a slow patch at the boundary where it was introduced, not three releases later when someone files a latency bug.

Core principle: a benchmark number is meaningless on its own. A single "after" run proves nothing. A verdict requires a baseline to compare against and a way to separate a real change from run-to-run noise. Everything below exists to enforce those two things, because the natural temptation is to run once and eyeball it — and that is worthless.

When to Use

• A backpressured loop is on a performance-sensitive project and needs to confirm a patch didn't regress perf — a quick targeted check each iteration, and the full suite before leaving the loop.

Not for: projects with no benchmark suite or no perf sensitivity — skip benchmarking entirely rather than inventing it. This is not a correctness check; tests cover that.

Step 1 — Discover the suites and their time budgets

If the project has a BACKPRESSURE.md, check it first — it may name the suite(s) and exact commands to use. Otherwise find the commands yourself — package.json scripts, Makefile, a benchmarks//bench/ dir, CI config, README. Identify which suites exist and roughly how long each takes. Expect a spread: fast micro/targeted suites (seconds) and full end-to-end suites (minutes). Note what each one actually measures so you can map a suite to the code path you touched.

Step 2 — Pick the suite for the moment

• Each iteration: run the fast, targeted suite that covers the path you changed. It must be cheap enough to run after every patch without rabbit-holing.
• Before done: run the full suite once. Targeted-green is necessary, not sufficient — a change can help one path and regress another.

Don't run an 8-minute end-to-end suite to check a change that only touches one micro path; don't declare done off a micro suite alone.

Step 3 — Always measure baseline vs patched

Never judge from the patched numbers alone. Run the unchanged baseline (git stash, or check out the parent commit / a worktree) and your change back-to-back, on the same machine under the same conditions. The baseline is the only thing that makes the "after" number mean anything.

Quiet the machine first — close background load, don't run benchmarks mid-build or while other heavy processes compete. Uncontrolled machine state is the largest real source of noise, and "same conditions" means actually the same, not just same hardware.

Step 4 — Beat the noise with repetition

Benchmark output is noisy (often ±3–5% run to run). One run per side is not evidence.

• Run each side multiple times (≥5 when the suite is cheap).
• Compare the median (and the min for CPU-bound microbenchmarks — least contended, most stable), not a single sample.
• Establish the noise band from the spread of the repeated runs (or a known per-suite figure). The band is your threshold for everything below.

Step 5 — The verdict: regression / improvement / wash

If a suspicious result still won't separate from noise after more runs — a small true shift that the suite genuinely can't resolve — do not default it to a pass. Treat it as a blocker and name it (the change may need a less noisy benchmark, or a human call), exactly as you would any other unresolved check.

Step 6 — Record the evidence

"No regression" asserted with no numbers is not a passing check. Record baseline vs patched, run count, and the median/min you compared. Evidence before assertion — the same standard the rest of the loop holds you to.

Common rationalizations

Red flags — STOP

• Reporting a benchmark result with no baseline to compare against.
• Declaring "no regression" with no recorded numbers.
• Calling an ambiguous straddle-the-band result a pass instead of adding runs.
• Skipping the full suite before done because a targeted suite was green.
• Treating a real, beyond-noise slowdown as acceptable without fixing it or naming the blocker.