001tmf/blatant-why/by-campaign-optimizer

BY Campaign Optimizer (Active Learning)

Multi-round design campaigns generate scored designs at every iteration. This skill turns that scoring history into actionable parameter changes for the next round — training a lightweight Random Forest on the designs you already have, ranking which features actually discriminate good from bad, and proposing thresholds and design counts that target the most promising regions of feature space.

The optimizer is the iteration loop: it runs after the diagnosis step (by-failure-diagnosis) has identified which features matter, and decides what parameters to push or relax for the next round. Diagnosis answers "what went wrong?"; this skill answers "what should we try next?".

When to Use This Skill

✅ Use this skill when:

• A design round has completed and scores are written to *_scores.json files
• The campaign has ≥10 scored designs total across all rounds
• You are deciding how to configure the next round of a multi-round campaign
• by-failure-diagnosis has identified discriminating features and you need

concrete threshold values

• You want a feature-importance ranking to decide what to optimize for
• The user asks "what should I change for the next round?" or "tune my campaign"

❌ Do NOT use this skill when:

• The campaign has fewer than 10 scored designs → use rule-based defaults from

by-campaign-manager instead

• This is the first round (no prior scores exist) → use by-research and

by-design-workflow to set initial parameters

• Designs are failing for non-statistical reasons (e.g., compute errors, bad

target structure) → run by-failure-diagnosis first to confirm the signal is real

• The campaign is a single-shot screen with no planned iterations
• You only need a "why did designs fail?" report → use by-failure-diagnosis

(the diagnostic counterpart to this optimizer)

Quick Start

from proteus_cli.campaign.active_learning import (
    has_enough_data,
    suggest_from_campaign,
)
campaign_dir = "campaigns/tnfa/campaign_20260520_001"
if has_enough_data(campaign_dir):
    result = suggest_from_campaign(campaign_dir)
    print(f"Source: {result.source}")                  # "active_learning"
    print(f"Confidence: {result.confidence}")          # "high" if >30 designs
    print(f"Top feature: {result.feature_importances[0]}")
    print(f"Recommendations: {result.recommended_parameters}")

Or via the CLI scripts shipped with this skill:

# Step 1: train RF, write optimizer_output.json
python scripts/optimize_from_csv.py \
    --scores campaign/round_1/scores.csv \
    --output campaign/round_1/optimizer_output.json
 
# Step 2: write next-round YAML config
python scripts/propose_next_round.py \
    --optimizer-output campaign/round_1/optimizer_output.json \
    --previous-config campaign/round_1/config.yaml \
    --output campaign/round_2/config.yaml

Installation

Or install the ML extras bundle:

pip install by-agent[ml]

License Compliance: All packages permit commercial use in AI applications.

Inputs

Required:

• Scored designs: either
• A campaign directory containing *_scores.json files (Python API), or
• A flat CSV with one row per design (CLI scripts)
• Each scored design must contain numeric values for some subset of:

ipsae (or ipsae_min), iptm, plddt, rmsd, liabilities, cdr3_length, net_charge, hydrophobic_fraction

• A status column (PASS / FAIL) is optional but improves recommendations

Optional:

• min_designs (default 10) — the active-learning threshold
• A previous-round YAML config to use as a template for the next round
• A target feature to optimize (defaults to ipsae)

See references/active-learning.md for sample-size guidance and references/random-forest-approach.md for feature-engineering details.

Outputs

Primary results:

• optimizer_output.json — feature importances, recommended thresholds,

exploration regions, confidence level, and a human-readable explanation

• OptimizationResult dataclass (Python API) with the same content

Next-round config:

• config_round_N+1.yaml — design parameters with adjusted thresholds,

updated num_designs, and feature targets carried over from the optimizer

Diagnostic fields in `optimizer_output.json`:

• source — "active_learning" if RF trained, "rule_based" if fallback
• confidence — "low" / "medium" / "high" based on training-set size
• feature_importances — sorted list of (feature, importance) tuples
• recommended_parameters — concrete values for the next round
• exploration_regions — feature-space regions to sample more densely
• warnings — any files skipped or features dropped

Clarification Questions

⚠️ CRITICAL: ASK THIS FIRST. Always confirm the user has at least 10 scored designs before running the optimizer. Below the threshold, the RF will not train and you should route to rule-based defaults instead.

1. Scored data available? (ASK THIS FIRST)

How many scored designs exist in the campaign? Where are the *_scores.json files (or the scores CSV)? If fewer than 10, stop and use rule-based defaults.

1. Target metric to optimize.

ipSAE is the default. Should we optimize for composite score, ipTM, or a custom weighted target? Different targets produce different recommendations.

1. Pass/fail labels.

Does the data include a status column with PASS/FAIL labels, or only continuous scores? With labels, the optimizer can suggest thresholds that match the empirical pass boundary.

1. Has diagnosis been run?

If by-failure-diagnosis flagged a feature as a strong discriminator, prioritize that feature when interpreting RF importances.

1. Compute and design-count budget for next round.

How many designs is the user willing to run? The optimizer can suggest increase_num_designs if the search space is under-sampled.

1. Diversity vs exploitation.

Should the next round exploit the top-quartile region (narrow thresholds) or explore widely (lower thresholds, higher diversity weight)? See references/active-learning.md#acquisition-functions.

1. Round number.

Which round number is being planned? The optimizer keeps a history file so later rounds can compare improvement trajectories.

Standard Workflow

🚨 MANDATORY: USE THE SHIPPED SCRIPTS — DO NOT REIMPLEMENT THE RF INLINE. 🚨

1. Confirm data sufficiency.

``python from proteus_cli.campaign.active_learning import has_enough_data assert has_enough_data(campaign_dir, min_designs=10) ` ✅ **VERIFICATION:** True printed. If False`, stop and use rule-based defaults.

1. (Optional) Run diagnosis first.

If the previous-round pass rate is below 20%, invoke by-failure-diagnosis to identify discriminating features. Use those features as priors when interpreting the optimizer's importance ranking.

1. Train RF and write optimizer output.

``bash python scripts/optimize_from_csv.py \ --scores campaign/round_N/scores.csv \ --target ipsae \ --output campaign/round_N/optimizer_output.json ` ✅ **VERIFICATION:** ✓ Optimizer completed: N designs, top feature: <feat>`

1. Inspect feature importances.

Read optimizer_output.json. The top feature drives the most variance in ipSAE; the recommended thresholds are derived from the top-quartile mean.

1. Propose next-round config.

``bash python scripts/propose_next_round.py \ --optimizer-output campaign/round_N/optimizer_output.json \ --previous-config campaign/round_N/config.yaml \ --output campaign/round_N+1/config.yaml ` ✅ **VERIFICATION:** ✓ Next-round config written: <path>`

1. Review the diff.

Compare the new config to the previous one. Surface every parameter change to the user before launching the next round.

1. Launch next round.

Hand off to by-design-workflow with the new config. The optimizer does not submit jobs.

❌ DON'T:

• ❌ Skip step 1 — running RF on <10 designs returns garbage importances
• ❌ Discard the optimizer's warnings list — it flags skipped files and dropped features
• ❌ Apply recommendations blindly when confidence is "low" — review with the user first

When Scripts Fail

Follow the standard Script Failure Hierarchy:

1. Fix and retry (90%) — Install scikit-learn / pyyaml; re-run.
2. Modify the script (5%) — Adjust --target or feature list if your

campaign uses non-standard metric names.

1. Use as reference (4%) — Read optimize_from_csv.py and adapt the

training loop if you need a different model class.

1. Write from scratch (1%) — Only if the campaign requires a

fundamentally different optimizer (e.g., Bayesian optimization with a Gaussian process).

For active-learning theory, see references/active-learning.md. For RF hyperparameter rationale, see references/random-forest-approach.md.

Decision Points

See references/active-learning.md#acquisition-functions for the full decision tree.

Common Issues