kohulan/orthonym-skills/refusal-census

Census the refusals

A run record alone cannot tell you why your pipeline abstained or failed. A common failure mode: every abstaining row carries the identical triple (e.g. tier=T5, source=abstain, reason=UNNAMEABLE) — one mechanism label, 100% of rows, zero resolution. You cannot rank build order from that alone, and prioritisation stalls until you can.

The information usually exists — it is just not in the run record. Most rule- or pipeline-based chemistry tools log a refusal/reject event per internal decision point (a template that didn't match, a candidate that failed a constraint, a ring system with no handler, a functional group with no rule, …) even when the summary metric collapses them all into one bucket. If your project has such logs and a parser for the codes in them, this skill connects the two. If it doesn't yet, build the parser first — usually a small regex/groupby over structured log lines, not new instrumentation.

The procedure

Copy this checklist and track your progress:

• [ ] 1. Per-row log captured on the corpus, split and tier the goal is defined over
• [ ] 2. Parsed to codes (reason optional), mapped to sites, deduplicated per input; ≥90% of

failing rows carry a code

• [ ] 3. Ranked by sole_blocker, with first_refusal_site and touched_sites beside it
• [ ] 4. Cross-tabbed against the cluster-failures clusters
• [ ] 5. Mean sites per failing input stated; corpus, tier, commit and row count cited

If Step 1 or 2 fails, report UNUSABLE (see Refusal conditions); once the capture is fixed, return to Step 1.

1. Capture the log, per row. Run your pipeline with its logger captured at WARNING-and-above (or whatever level carries refusal events), one record per input. If a hung input can hang the whole batch, run in short-lived worker subprocesses with a per-item timeout so one bad input costs one slot, not the whole run.

2. Parse to codes with your refusal-code parser. Three behaviours worth building in deliberately, seen to matter in practice:

• treat any structured "reason" field as optional — real sites often log a bare code

with no reason attached, or one with a non-numeric suffix. Requiring a well-formed reason string silently erases those sites from the census.

• deduplicate per input. A code firing five times while the pipeline probes five

substructures/candidates for one input is still one blocker for that input. The question is "how many inputs does this site block", not "how often does it fire" — un-deduplicated counts mis-rank the build order.

• map codes to sites. A site that logs several codes counts once per input.

3. Rank by sole blocker; report the other two rankings beside it.

These differ substantially and answer different questions. (Illustrative shape from a real census: one site was the first-fired blocker on roughly 55% of failing inputs and touched about 68% of them; a second site touched more inputs overall — about 64% — but led on far fewer.) Build in sole-blocker order, because only those inputs flip when that one site is fixed. Keep first_refusal_site and touched_sites as extra information: the first-refusal count names the proximate blocker, and the touched count tells you what else will still block those same inputs afterwards, so both set expectations, not build order.

4. Cross-tab against the structural feature clusters from cluster-failures — scaffold class, charge state, stereocenter count, size band, functional-group class. A site that blocks one cluster is a different piece of work from a site that blocks all of them. Also split each site's count into candidate-availability (nothing valid was generated) and candidate-selection (a valid candidate lost); they need different fixes.

5. Report the mean sites per failing input. If it's, say, 3–4 (median 4), most failures are blocked at several sites simultaneously, so fixing the leader alone will not flip most of them to success (its sole-blocker count is how many it does flip). Ranking without this number over-promises the win — say it up front, before anyone commits to a build order.

Refusal conditions

Report UNUSABLE and stop if:

• fewer than ~90% of failing rows carry ≥1 code — the capture is broken, not the

pipeline.

• the census was run on a different corpus, split, or tier than the one the goal is

defined over. This is a real trap: a census computed on one dataset/commit/tier does not transfer to another — re-measure on the target split before sizing any build against it.

Related

• check-target — validate the class before scoping it.
• spy-site — prove the site is actually on the execution path before editing it.
• cluster-failures — the structural feature clusters to cross-tab against.
• reuse-before-rerun — look for an existing census of the same corpus, tier and commit

before capturing a new one.

• Your project's own fix-methodology / engineering-log doc — verify the capture/harness

actually ran before trusting any percentage (a silently-broken capture reports a fake 100% or a fake 0% just as convincingly as a real one).