001tmf/blatant-why/by-experiment-results

BY Experiment Results Skill

A design campaign only learns when lab outcomes are joined back to the in-silico predictions that produced them. This skill is the wet-lab feedback loop closer: it ingests lab readouts (Adaptyv batch CSVs, internal ELISA plate-reader output, BLI/Octet kinetics, expression QC), joins them to the per-design feature table the screener emitted, and runs a calibration analysis that asks one question for every in-silico feature:

Did this feature actually predict whether the design worked in the lab?

The answer for each feature is one of: validated (in-silico PASS correlated with lab PASS), contradicted (predictor pointed the opposite direction), or inconclusive (no signal). The diagnosis goes into the knowledge graph with that confidence label so future campaigns start from real-world-calibrated priors, not silicon-only priors.

This skill sits between by-screening (which produces in-silico PASS/FAIL) and by-campaign-optimizer (which trains the next round). It is the only skill in BY that consumes ground-truth lab data.

When to Use This Skill

✅ Use this skill when:

• A lab readout file (CSV or Excel) has arrived from Adaptyv Bio for a previously submitted batch
• An internal ELISA, BLI, or Octet run has produced a tidy results table for designs the BY pipeline scored
• You need to validate round-N in-silico predictions against round-N-1 lab outcomes
• A user asks "did our predictions hold up?", "calibrate the screener", or "did ipSAE correlate with Kd?"
• A round of designs has come back from the lab and you need to update the campaign-optimizer training data with ground truth
• You want to write a calibration report showing precision at top-K and lift over random
• You want to publish findings to the knowledge graph as validated or contradicted with confidence levels

❌ Don't use this skill for:

• A campaign that has not yet been submitted to lab — there is no ground truth to compare. Use by-screening + by-failure-diagnosis first.
• Pre-submission ranking of designs — that is by-screening and by-design-workflow.
• Computing raw in-silico scores from PDB / NPZ — use by-scoring.
• Diagnosing why a campaign produced low in-silico pass rate — use by-failure-diagnosis (no lab data needed).
• Lab submission itself — use the gated /by:approve-lab flow and by-adaptyv MCP tools.
• Curating raw structural files (CIF / PDB) — those stay in the campaign directory.

Quick Start

The shortest path from a lab CSV to a calibration report and knowledge-graph update is three CLI calls:

# 1) Normalize the lab readout to canonical schema
python3 scripts/ingest_lab_results.py \
    --input  experiments/batch_001_adaptyv.csv \
    --output campaigns/<target>/<campaign_id>/lab_results.normalized.json \
    --assay  affinity_bli \
    --aggregate median
 
# 2) Join with in-silico features and run calibration
python3 scripts/diagnose_silico_vs_lab.py \
    --lab          campaigns/<target>/<campaign_id>/lab_results.normalized.json \
    --silico       campaigns/<target>/<campaign_id>/screening_results.json \
    --output-json  campaigns/<target>/<campaign_id>/calibration.json \
    --output-md    campaigns/<target>/<campaign_id>/calibration_report.md
 
# 3) Push validated / contradicted findings to the knowledge graph
python3 scripts/update_knowledge_from_lab.py \
    --calibration campaigns/<target>/<campaign_id>/calibration.json \
    --campaign-id <campaign_id> \
    --target      <target>

✅ VERIFICATION: calibration.json contains a features[] array where each feature has a verdict of validated, contradicted, or inconclusive, and calibration_report.md opens with a one-line headline like "12 / 18 lab-tested designs bound (67%); ipSAE_min was the strongest validated predictor (p=0.004, AUC=0.81)."

Installation

License Compliance: All third-party packages permit commercial use in AI applications.

System requirements: CPU-only; no GPU; no internet (unless pulling Adaptyv results via the MCP tool). All scripts run in seconds for batches up to ~1,000 designs.

Inputs

Required:

• Lab readout file (CSV or Excel; .csv / .tsv / .xlsx):
• Must contain a design_id column (canonical) or a renameable alias mapped via --alias design_id=<src>. The design_id MUST match the identifiers used in the campaign's screening_results.json.
• Must contain a ground-truth outcome column. Acceptable shapes:
• Categorical: lab_outcome ∈ {PASS, FAIL} or {binder, no-binder} — set via --outcome-col and --outcome-pass-value.
• Continuous Kd (nM): kd_nm with a --kd-threshold-nm flag (default 100 nM) to derive PASS/FAIL.
• Continuous binding signal: binding_signal (a.u.) with --binding-threshold flag.
• Optional columns: ka_per_M_per_s, kd_per_s, expression_mg_per_L, polyspecificity_score, aggregation_pct, replicate_id.
• In-silico feature file (screening_results.json from by-screening, or a CSV):
• One row per design, with a design_id column and any subset of: ipsae, ipsae_min, iptm, plddt, rmsd, liabilities, cdr3_length, net_charge, hydrophobic_fraction, aggregation_predicted, polyspecificity_predicted.
• Assay type (CLI arg --assay): one of affinity_elisa, affinity_bli, affinity_octet, expression_qc, polyspecificity_panel. Determines which canonical columns are expected and how the calibration metrics are interpreted.

Alternative inputs:

• Adaptyv API — fetch results via mcp__by-adaptyv__adaptyv_get_results(experiment_id=...), save the response, then feed to ingest_lab_results.py --input <saved.json> --format adaptyv-json.
• Multi-batch CSV — pass --batch-col batch to keep batches separate when aggregating replicates.

Optional:

• Replicate aggregation policy (--aggregate): one of mean, median, keep-all (default median for affinity, mean for signal). Replicates are aggregated on (design_id, replicate_id).
• Column aliases (--alias canonical=source, repeatable): handles lab-batch column drift (e.g. --alias design_id=variant_name --alias kd_nm=KD_nM).
• Censored-data threshold (--lod-kd-nm): values reported as > LOD are treated as right-censored and excluded from continuous tests (but still kept as FAIL for categorical).

See references/lab-readout-schemas.md for column-rename tables for each lab partner and assay.

Outputs

Primary results:

• lab_results.normalized.json — canonical schema lab data (one record per design after aggregation), with provenance fields (source_file, aggregation_policy, assay, batch_id).
• enriched_dataset.csv — wide table joining the normalized lab outcomes to every in-silico feature on design_id. This is the training data that by-campaign-optimizer picks up for the next round.
• calibration.json — per-feature verdicts:
• features[].name, .verdict (validated / contradicted / inconclusive)
• .statistic (Mann-Whitney U), .p_value, .q_value (BH-corrected), .effect_size
• .passed_mean, .failed_mean (in-lab groups)
• .precision_at_top_k, .lift_over_random, .auc (if classifier-style)
• .interpretation — one-line natural language summary
• calibration_report.md — human-readable report with headline, top validated predictors, top contradicted predictors, and a recommended action for the next round.

Updates to the knowledge graph (from update_knowledge_from_lab.py):

• One knowledge_store_campaign record with the lab outcomes appended to the outcomes dict and the joined designs[] carrying both in-silico and lab fields.
• One knowledge_store_failure record per contradicted feature (so future campaigns know not to trust that predictor for similar targets).

Persistence:

• Write calibration.json next to screening_results.json in the campaign directory so by-campaign-optimizer can pick it up without further configuration.

Clarification Questions

⚠️ CRITICAL: Always ask question #1 first to confirm the lab file exists and the ground-truth column is identified before any compute runs.

1. Lab readout file and outcome column (ASK THIS FIRST):

• Where is the lab readout file (path)? What format (CSV, TSV, Excel, Adaptyv JSON)?
• Which column holds the ground-truth outcome? Is it categorical (PASS/FAIL, binder/no-binder) or continuous (Kd in nM, binding signal a.u.)?
• Without confirmed outcome semantics, the calibration is meaningless. Stop here if uncertain.

1. Design ID alignment:

• Does the design_id column in the lab file match the design IDs in screening_results.json exactly? Lab partners often re-encode IDs (e.g. BY-001 ↔ by_001 ↔ BY001). Decide an alias map before running ingest.

1. Assay type:

• Affinity (ELISA, BLI, Octet), expression QC, or polyspecificity panel? Different assays have different canonical columns and different PASS thresholds.

1. Replicate handling:

• Are there technical replicates per design? Do we aggregate by median (default for Kd), mean (default for signal), or keep all rows for downstream variance analysis?

1. Continuous → categorical threshold:

• If the outcome is continuous (Kd, signal), what threshold defines PASS? Default: Kd ≤ 100 nM = PASS. The user should confirm — every target has a different therapeutic-relevance bar.

1. Censored data:

• Were any measurements reported as > LOD (above limit of detection)? These are right-censored — treat as FAIL for categorical analysis, exclude from continuous tests.

1. Knowledge-graph write authority:

• Should we push findings to the knowledge graph now (default: yes), or stage them for review first? Findings are append-only and influence future campaigns.

For a fuller pre-flight when the lab file is from a new vendor, walk through references/lab-readout-schemas.md.

Standard Workflow

🚨 MANDATORY: USE THE SHIPPED SCRIPTS — DO NOT REIMPLEMENT THE JOIN OR STATISTICS INLINE 🚨

Inline joins on design_id invariably mis-handle column drift, replicate aggregation, censored values, and BH correction. The shipped scripts encode all of these consistently with by-failure-diagnosis (same Mann-Whitney + BH approach), so calibration numbers stay comparable across rounds.

Step 1: Verify file presence and outcome semantics

Before running anything, confirm the lab file exists and you know which column holds the ground truth.

ls -lh experiments/batch_001_adaptyv.csv
head -3 experiments/batch_001_adaptyv.csv

✅ VERIFICATION: File is non-empty; you can identify a design_id column and an outcome column.

Step 2: Normalize the lab readout

python3 scripts/ingest_lab_results.py \
    --input  experiments/batch_001_adaptyv.csv \
    --output campaigns/<target>/<campaign_id>/lab_results.normalized.json \
    --assay  affinity_bli \
    --aggregate median \
    --kd-threshold-nm 100

If the lab CSV uses non-canonical column names, pass aliases: