001tmf/blatant-why/by-failure-diagnosis

BY Failure Diagnosis Skill

Closing the design feedback loop requires understanding why designs fail, not just that they fail. This skill compares the distribution of every continuous feature between PASS and FAIL designs using non-parametric statistics, ranks features by discriminating power, and translates the result into concrete threshold or campaign-parameter changes for the next iteration.

It is the bridge between screening (which produces PASS/FAIL labels) and campaign optimization (which adjusts parameters for the next round).

When to Use This Skill

Use this skill when:

• ✅ Pass rate is below 20% in a screening round and you need to know why before re-spending compute
• ✅ At least 30 designs have been scored with a status field (PASS or FAIL)
• ✅ You have numeric features per design (ipSAE, ipTM, pLDDT, RMSD, liabilities, net_charge, hydrophobic_fraction, cdr3_length)
• ✅ User explicitly asks "why are my designs failing?", "diagnose failures", "what's going wrong?"
• ✅ Before the active-learning step in a multi-round campaign (route diagnosis → optimizer)
• ✅ After a screening regression where pass rate dropped versus a prior round

Don't use this skill when:

• ❌ Fewer than 30 designs total — statistical power is too low; the test will be noisy. Score more designs first.
• ❌ No FAIL designs (100% pass rate) — there is nothing to compare against. Either tighten thresholds or move to lab submission.
• ❌ No PASS designs (0% pass rate) — there is nothing to compare against. Use by-hypothesis-debate to pick a new strategy before spending more compute.
• ❌ You want to redesign individual residues — that is per-design rationale, not population statistics. Use by-epitope-analysis instead.
• ❌ You want to predict structures or score new designs — use protenix or by-scoring instead.
• ❌ The campaign has different scoring criteria across rounds — comparing apples to oranges; run diagnosis within a single round only.

Quick Start

The most common invocation: run the MCP tool against a campaign's screening output.

import json
# Load campaign screening results (one dict per design)
with open("campaigns/<target>/<campaign_id>/screening_results.json") as fh:
    designs = json.load(fh)
# Call the MCP tool with the array serialized as JSON
result = mcp__by-screening__screen_diagnose_failures(
    scores_json=json.dumps(designs),
    pass_key="status",
    pass_value="PASS",
)
# Parse and present
diag = json.loads(result)
print(diag["formatted"])           # human-readable table
print(diag["recommendations"])     # top-3 actionable suggestions

✅ VERIFICATION: Expect output like Failure Diagnosis (12/80 passed, 15% rate) followed by a table sorted by p-value, with effect sizes and recommendations.

For ad-hoc CSV-based diagnosis (no MCP server required), use the included scripts:

python3 scripts/diagnose_from_csv.py --input designs.csv --status-col status
python3 scripts/plot_distributions.py --input designs.csv --output diagnostics.pdf --top-n 5

Installation

License Compliance: All third-party packages permit commercial use in AI applications.

System requirements: No GPU needed. Diagnosis runs CPU-only in seconds for up to 10,000 designs. No internet required.

Inputs

Required:

• Designs array (or CSV) with one row per design:
• A status column (default name status) with values PASS or FAIL (uppercase)
• At least one numeric feature column from this set: ipsae, ipsae_min, iptm, plddt, rmsd, net_charge, hydrophobic_fraction, liabilities, cdr3_length
• Minimum 3 designs in each of PASS and FAIL groups (statistical floor)
• Recommended minimum 30 designs total (≥10 in each group) for usable power

Alternative inputs:

• JSON array (one object per design) — pass as scores_json to the MCP tool
• pandas DataFrame — convert to records with df.to_dict(orient="records") before passing
• CSV file — use scripts/diagnose_from_csv.py directly

Optional:

• `pass_key` — column name for status (default status)
• `pass_value` — value indicating PASS (default PASS)
• Extra feature columns — the MCP tool only tests the canonical 9 features; custom features require the CSV script (see scripts/diagnose_from_csv.py --help)

See references/failure-patterns.md for which features matter for which failure modes.

Outputs

Primary results (JSON object from MCP tool):

• total_designs, passed, failed, pass_rate — campaign-level counts
• discriminating_features[] — sorted by p-value ascending, each entry has:
• feature_name, test_type (mann_whitney), statistic, p_value, effect_size
• passed_mean, failed_mean — group means
• interpretation — one-line natural language summary
• summary — short headline string
• recommendations[] — up to 3 actionable threshold/parameter suggestions
• formatted — human-readable text table

Visualizations (from `plot_distributions.py`):

• diagnostics.pdf — single multi-panel PDF with violin plots of PASS vs FAIL for each significant feature (300 DPI)

Reports (from `diagnose_from_csv.py`):

• stdout — formatted markdown-style table sorted by p-value, with BH-corrected q-values

Analysis objects:

• The MCP tool returns a JSON string; persist it as campaigns/<target>/<campaign_id>/diagnosis.json for downstream consumption by by-campaign-optimizer and by-knowledge.

Clarification Questions

⚠️ CRITICAL: Always ask question #1 first to confirm input data exists.

1. Input data (ASK THIS FIRST):

• Do you have a CSV, JSON file, or campaign directory with per-design scores plus a PASS/FAIL status column?
• If no — run by-screening first to produce labels.

1. Group sizes:

• How many designs are in PASS vs FAIL? If either is below 3, diagnosis cannot run. If either is below 10, results will be noisy.

1. Status column convention:

• Is the column named status? Are values PASS/FAIL (uppercase)? If different, override with pass_key and pass_value.

1. Features available:

• Which of the canonical 9 features are present? Diagnosis silently skips missing or all-null columns. Confirm at least 2 are present.

1. Round scope:

• Is this a single screening round, or are you mixing rounds with different thresholds? Mixing rounds invalidates the test. Filter to one round first.

1. Downstream intent:

• Will you adjust screening thresholds, change campaign parameters (scaffolds, hotspots), or both? This affects which recommendations to surface in the report.

1. Primary objective:

• Are you trying to (a) understand the failure mode, (b) tune thresholds for the same model, or (c) decide whether to switch strategy entirely? See references/threshold-tuning.md for mapping diagnosis output to each action.

For detailed clarification flow when working with a campaign directory, read references/failure-patterns.md.

Standard Workflow

🚨 MANDATORY: USE THE MCP TOOL OR PROVIDED SCRIPTS - DO NOT WRITE INLINE STATISTICS 🚨

This skill enforces a single statistical methodology (Mann-Whitney U with Benjamini-Hochberg correction). Re-implementing the test inline risks silent statistical errors (wrong tails, missing tie correction, no multiple-testing adjustment).

Step 1: Collect designs and confirm group sizes

Load the campaign's screening output. Confirm at least 3 designs in each of PASS and FAIL.

import json
designs = json.load(open("campaigns/<target>/<id>/screening_results.json"))
passed = sum(1 for d in designs if d.get("status") == "PASS")
failed = len(designs) - passed
assert passed >= 3 and failed >= 3, "Need ≥3 in each group"

✅ VERIFICATION: Print counts: PASS=12, FAIL=68 (total=80, rate=15.0%). If pass rate ≥ 20%, ask the user whether diagnosis is still desired.

Step 2: Call the MCP tool

result_str = mcp__by-screening__screen_diagnose_failures(
    scores_json=json.dumps(designs),
    pass_key="status",
    pass_value="PASS",
)
diag = json.loads(result_str)

✅ VERIFICATION: diag["summary"] is a non-empty string. diag["discriminating_features"] is a list of length ≥ 0.

Step 3: Interpret the table

Sort already done by the tool (ascending p-value). Apply the interpretation cheat-sheet:

p-value caveats: With 9 features tested, expect ~0.45 false positives at α=0.05. Always check the BH-corrected q-values from scripts/diagnose_from_csv.py before declaring a feature significant. See references/mann-whitney-guide.md for the full math.

Step 4: Plot distributions (optional but recommended)

For any feature with p < 0.05 and effect size > 0.5, plot the PASS vs FAIL distribution:

python3 scripts/plot_distributions.py \
    --input campaigns/<target>/<id>/screening_results.csv \
    --output campaigns/<target>/<id>/diagnostics.pdf \
    --top-n 5 \
    --status-col status

✅ VERIFICATION: diagnostics.pdf exists; each panel shows two violins (PASS, FAIL) with median lines.

Step 5: Translate to action

Match the diagnosis pattern against references/threshold-tuning.md. Common moves:

• Single feature dominates (e.g., ipsae effect > 1.5, others < 0.5) → raise that feature's threshold
• Multiple features moderately correlated (CDR3 length + hydrophobic fraction) → re-run BoltzGen with constrained CDR3 length
• No significant features but low pass rate → the failure mode is not in your feature set; expand features or change strategy

Step 6: Persist and route

Write diagnosis.json to the campaign directory. Hand off to by-campaign-optimizer with the diagnosis as input. Append the diagnosis summary to by-knowledge for cross-campaign learning.

with open("campaigns/<target>/<id>/diagnosis.json", "w") as fh:
    json.dump(diag, fh, indent=2)

⚠️ CRITICAL - DO NOT:

• ❌ Run the test with mixed rounds → STOP: filter to one round first
• ❌ Use a t-test instead of Mann-Whitney → screening features are non-normal
• ❌ Trust a single feature with p < 0.05 but effect size < 0.3 → almost certainly a false positive
• ❌ Skip the BH correction when reporting > 1 feature → inflated false-positive rate

When Scripts Fail

Use the standard escalation hierarchy: