mazebench/mazebenchengine/optimize-with-environments

Optimize With Environments

Goal

Use GEPA to optimize system prompts in a controlled, reproducible loop.

Scope

Current GEPA path is for system prompt optimization. If user asks for unsupported optimization targets, stop and clarify before proceeding.

Endpoint And Model Selection Nudge

1. Encourage users to define reusable aliases in configs/endpoints.toml.
2. Ask whether optimization should be validated on instruct or reasoning models.
3. Instruct go-tos: gpt-4.1 series, qwen3 instruct series.
4. Reasoning go-tos: gpt-5 series, qwen3 thinking series, glm series.
5. For benchmark reporting, keep model family fixed between baseline and optimized comparisons unless the user requests a cross-family study.
6. Endpoint entries support optional headers (or extra_headers) for custom HTTP headers. GEPA inherits these from the registry for both the main model and the reflection model:

[[endpoint]]
endpoint_id = "my-proxy"
model = "gpt-4.1-mini"
url = "https://api.example/v1"
key = "OPENAI_API_KEY"
headers = { "X-Custom-Header" = "value" }

Core Workflow

1. Verify baseline first with prime eval run. Keep the default save behavior and do not add --skip-upload unless the user explicitly requests that deviation:

prime eval run my-env -m openai/gpt-4.1-mini -n 50 -r 3 -s

1. For v1 Taskset + Harness environments, confirm prompt-like fields are exposed in the saved state or task info before GEPA reflection; BYO Harness implementations may render richer trajectories than classic MultiTurnEnv examples.
2. Run GEPA:

prime gepa run my-env -m openai/gpt-4.1-mini -M openai/gpt-4.1-mini -B 500 -n 100 -N 50

1. Keep v1 environment settings under taskset and harness config sections:

[[env]]
id = "my-env"
 
[env.taskset]
split = "train"
 
[env.harness]
max_turns = 8

1. Or run from config:

prime gepa run configs/gepa/qwen-3-5.toml

1. Re-evaluate with optimized prompt and compare against baseline.

High-Value Settings

1. -B/--max-calls: total optimization budget.
2. -n/--num-train and -N/--num-val: train/validation split sizes.
3. --minibatch-size: reflection granularity.
4. --perfect-score: skip already-solved minibatches when max score is known.
5. --state-columns: include environment-specific context in reflection data.

Output Artifacts

Expect and inspect:

1. system_prompt.txt
2. pareto_frontier.jsonl
3. metadata.json

Load optimized prompts with vf.SystemMessage.from_path("/path/to/system_prompt.txt") so the saved prompt is used verbatim.

Quality Rules

1. Do not optimize on top of broken reward logic.
2. For weak deterministic checks, fix rubric quality before GEPA tuning.
3. Keep model, sampling, and dataset conditions stable during baseline-vs-GEPA comparison.
4. Report limitations directly when feature gaps block requested optimization.

Deliverable

Return:

1. Baseline metrics.
2. Optimized metrics.
3. Prompt diff summary.
4. Recommendation to adopt, iterate, or stop.