001tmf/blatant-why/by-hypothesis-debate

BY Hypothesis-Debate Skill

Picking the wrong design strategy on a novel target wastes a GPU day. This skill forces an explicit, structured debate before any compute is spent: three hypothesis agents propose competing strategies in parallel, then a reflection agent ranks them against a fixed rubric. The winner becomes the campaign config.

The pattern is borrowed from adversarial ML and red-team review — it works because three independently-constrained agents will surface failure modes that a single planner misses, and the reflection step exposes hidden assumptions that would otherwise propagate into the campaign plan.

When to Use This Skill

✅ Use this skill when:

• Starting a campaign against a novel target (0–1 PDB structures, no SAbDab antibodies, <10 papers)
• Research findings have CONTRADICTED confidence (sources disagree on epitope, modality, or scaffold)
• Multiple viable modalities are on the table (VHH vs scFv vs de novo binder) and the user has not pinned one
• The user explicitly says "compare approaches", "explore options", "which strategy would you pick"
• Target difficulty is rated novel or exploratory by by-research
• A previous campaign against the same target failed with hit rate < 5% (this skill helps replan, not retry)

❌ Do NOT use this skill when:

• Target is well-studied with a clear precedent (e.g., TNF-alpha, PD-L1, HER2) → use by-campaign-manager directly
• The user has already specified modality, scaffolds, and tier — debating wastes turns
• Running a preview / feasibility campaign — overhead of 3-agent debate is not worth it for 5–10 designs
• Iterating on a previously-approved campaign (round 2, 3, … reuse the prior strategy)
• The campaign is gated on lab submission — debate occurs upstream of /by:approve-lab, not after

Quick Start

Invoke from the orchestrator (main session) after by-research has written research.md and recommended_hotspots.json to the campaign directory:

python3 scripts/orchestrate_debate.py \
  --research-dir campaigns/IL23R/campaign_20260520_001/research \
  --campaign-config campaigns/IL23R/campaign_20260520_001/campaign_config.yaml \
  --output-dir campaigns/IL23R/campaign_20260520_001/debate

Expected runtime: 4–8 minutes (3 hypothesis agents run in parallel ~3 min each, reflection agent runs sequentially ~2 min). Expected output: debate/proposals/ with three strategy_proposal.json files, debate/ranking.json, and an updated campaign_config.yaml reflecting the winning strategy.

Inputs

Required:

• Research report directory — must contain research.md, recommended_hotspots.json, design_recommendation.json, and validated_findings.json from by-research (8-phase pipeline)
• Campaign config skeleton — a YAML file with target, pdb_id, compute block, but modality, scaffolds, tier are TBD

Alternative inputs:

• `campaign_context.json` — if present (from /by:plan-campaign), the user's preferences seed the agents but do NOT override the debate. Agents see the preference as a prior, not a constraint
• `previous_debate.json` — if rerunning after a failed campaign, prior debate gives historical signal to the reflection agent

Optional:

• Debate budget — number of hypothesis agents (default 3; 2 acceptable for tight token budgets, 4–5 for high-stakes campaigns)
• Reflection temperature — default 0.2 for ranking stability; raise to 0.5 for genuinely novel targets to encourage challenging the rubric
• Tie-break mode — merge (default), aggressive_wins, user_decides

See references/agent-profiles.md for full input specifications including agent-specific directive templates.

Outputs

All outputs land under <campaign_dir>/debate/:

strategy_proposal.json Schema

Every hypothesis agent writes one of these. Validation: scripts/validate_proposal.py.

{
  "agent": "conservative",
  "version": "1.0",
  "target": "IL23R",
  "modality": "VHH",
  "protocol": "nanobody-anything",
  "scaffolds": ["caplacizumab", "ozoralizumab"],
  "tier": "standard",
  "num_designs_per_scaffold": 5000,
  "compute_provider": "local",
  "epitope": {
    "pdb_id": "6WDQ",
    "target_chain": "A",
    "residues": [78, 79, 80, 112, 113, 114, 115],
    "range_notation": "A78-A80,A112-A115"
  },
  "rationale": "Well-validated VHH scaffolds + low-novelty epitope reduces risk. Expected hit rate 20-40% based on prior IL-family campaigns.",
  "expected_hit_rate": "20-40%",
  "expected_wall_time_hours": 8,
  "key_risks": [
    "VHH may sterically clash with N-glycan at N91",
    "ipSAE may underestimate cytokine-receptor interface quality"
  ],
  "mitigation": "Run Protenix 5-seed ensemble validation on top 10 designs",
  "confidence": 0.78,
  "research_source_ids": ["src_001", "src_003", "src_007"]
}

ranking.json Schema

The reflection agent writes one per debate.

{
  "version": "1.0",
  "target": "IL23R",
  "ranked_at": "2026-05-20T14:32:00Z",
  "candidates": [
    {
      "agent": "conservative",
      "scientific_rigor": 0.85,
      "feasibility": 0.90,
      "innovation": 0.40,
      "risk_adjusted_confidence": 0.72,
      "composite_score": 0.7625,
      "rank": 1
    },
    {
      "agent": "diverse",
      "scientific_rigor": 0.70,
      "feasibility": 0.75,
      "innovation": 0.70,
      "risk_adjusted_confidence": 0.65,
      "composite_score": 0.7100,
      "rank": 2
    },
    {
      "agent": "aggressive",
      "scientific_rigor": 0.55,
      "feasibility": 0.50,
      "innovation": 0.95,
      "risk_adjusted_confidence": 0.45,
      "composite_score": 0.6125,
      "rank": 3
    }
  ],
  "winner": "conservative",
  "tie_break_applied": false,
  "rationale": "Conservative strategy wins on feasibility (0.90) and rigor (0.85); Diverse came close but lower research-source coverage.",
  "dissenting_notes": "Aggressive proposal flagged a real concern about VHH glycan clash — fold this into the conservative plan's mitigation.",
  "merged_recommendations": [
    "Add 'cross_check_glycan' flag from aggressive proposal to winning config",
    "Reserve 1500 designs from diverse proposal for alt-epitope exploration in run_002"
  ]
}

For the full schema with field-by-field type constraints, see references/ranking-rubric.md.

Clarification Questions

⚠️ CRITICAL: ASK THIS FIRST — confirm research is complete before debating.

1. Research output present? (ASK THIS FIRST)

• Does <campaign_dir>/research/research.md exist? If not → run by-research first; debate is pointless without target context
• Are there recommended_hotspots.json AND design_recommendation.json? If not → debate has no input
• If validated_findings.json shows zero HIGH confidence findings → tell user; the debate will likely produce low-confidence proposals

1. Has the user pinned a modality?

• If campaign_context.json specifies modality → seed the agents but allow them to challenge it
• If user said "use VHH" in conversation → still run debate; agents may surface a reason VHH is wrong here
• If user said "VHH only, do not consider alternatives" → SKIP debate, go straight to by-campaign-manager

1. What is the compute budget?

• Local GPU available? → all proposals can use compute_provider: local
• HPC (RunPod) only? → agents must size designs against HPC time budget
• Tamarind only? → agents must consider per-design cost in feasibility scoring

1. How novel is the target really?

• 0 PDB, 0 SAbDab, <5 papers → use full 3-agent debate, raise reflection temperature
• 0 PDB, 0 SAbDab, but has homologs with known binders → run debate; the Conservative agent will lean on homologs
• Has PDB but no antibodies → likely doesn't need debate; consider skipping

1. Tie-break preference?

• Default merge — winner gets best ideas folded in from runners-up
• aggressive_wins — when user explicitly wants the highest-novelty option to break ties
• user_decides — escalate to user when top 2 scores differ by < 5%

1. Is this a replan after failure?

• If yes, point to campaigns/<target>/.../previous_debate.json. Reflection agent will downrank strategies similar to the failed one
• Add the failure mode (e.g., "all designs had ipSAE < 0.4") to the orchestration prompt so agents avoid it

1. What is the primary goal of this campaign?

• Hit rate → favors Conservative (well-trodden paths)
• Diversity → favors Diverse (broad epitope/scaffold coverage)
• Novelty / first-in-class → favors Aggressive (high-novelty hypotheses)
• Balanced → reflection rubric weights all four dimensions equally

For decision detail and example transcripts, see references/agent-profiles.md §2.

Standard Workflow

🚨 MANDATORY: USE `scripts/orchestrate_debate.py` — DO NOT INLINE THE TASK() CALLS 🚨

The orchestrator script wraps Task() dispatch, agent prompt construction, JSON validation, and writes the debate log. Inlining the calls breaks the audit trail and skips proposal schema validation.

Step 1: Verify Research Inputs

ls <campaign_dir>/research/
# Must show: research.md, recommended_hotspots.json,
#            design_recommendation.json, validated_findings.json

✅ VERIFICATION: All four files exist and validated_findings.json has at least 1 finding with confidence HIGH or MEDIUM.

❌ DON'T: Start the debate if research is incomplete — agents will hallucinate.

Step 2: Run the Orchestrator

python3 scripts/orchestrate_debate.py \
  --research-dir <campaign_dir>/research \
  --campaign-config <campaign_dir>/campaign_config.yaml \
  --output-dir <campaign_dir>/debate \
  --num-agents 3 \
  --tie-break merge

Expected stdout:

✓ Loaded research from <campaign_dir>/research
✓ Spawned conservative agent (task_id=tsk_001)
✓ Spawned aggressive agent (task_id=tsk_002)
✓ Spawned diverse agent (task_id=tsk_003)
✓ Collected 3/3 proposals
✓ Validated all proposals against schema
✓ Spawned reflection agent (task_id=tsk_004)
✓ Ranking complete: winner=conservative, composite=0.7625
✓ Updated campaign_config.yaml
✓ Debate completed: 7 minutes 12 seconds

Step 3: Validate Proposals (Automatic)

The orchestrator calls scripts/validate_proposal.py on each proposal before passing to the reflection agent. If a proposal fails validation, it is dropped and reported. The reflection agent never sees malformed proposals.

Step 4: Review the Decision

Read <campaign_dir>/debate/decision_summary.md. Present the winner to the user with the dissenting notes from runners-up.

Step 5: Hand Off to by-campaign-manager

Once the user approves the winner, the updated campaign_config.yaml flows directly into by-campaign-manager for execution. No re-planning needed.

Script Failure Hierarchy