cyrusnuevodia/gepa-research/discover

Discover

Internal procedure for gepa-research:discover. The user only sees the user-facing prompts, the dashboard URL, and the baseline score -- everything else is the agent's choreography.

Host conventions

This skill runs on any host that implements the Agent Skills spec. When the body uses generic phrases, apply the host's best-fit equivalent:

• "ask the user" -- use your host's structured multi-choice question tool if you have one (e.g. AskUserQuestion, request_user_input). If the host has none, phrase the question as plain text in your next reply and wait for the user's answer.
• File paths like `references/...` -- relative to this SKILL.md; resolve from the skill directory.
• Slash commands shown in user-facing copy (e.g. /gepa-research:discover) -- translate to your host's mention syntax when speaking to the user (e.g. $gepa-research discover on Codex -- plugin namespace then skill name, separated by a space).

0. Verify the gepa-research CLI is available and in sync with the plugin

Before anything else, run:

gepa-research-version-check

This wraps gepa-research --version and additionally asserts the installed CLI matches the plugin manifest version (hosts refetch the plugin on version bumps, but do not reinstall the globally-installed CLI -- drift between the two breaks skills silently).

Four outcomes to handle:

1. Exit 0, `gepa-research-version-check: OK (plugin=X, cli=X)` -- continue to step 1.
2. Exit 1, "plugin manifest and installed CLI disagree" -- stop and show the user the script's stderr verbatim; it tells them the uv tool install --force "git+https://github.com/CyrusNuevoDia/gepa-research@v<version>#subdirectory=plugins/gepa-research" command to run. Then re-invoke this skill.
3. Exit 2, "gepa-research CLI not on PATH" -- stop and tell the user:

> gepa-research-cli isn't on your PATH. Install it once from GitHub: uv tool install "git+https://github.com/CyrusNuevoDia/gepa-research#subdirectory=plugins/gepa-research" (or substitute pipx install for uv tool install). Then re-invoke this skill.

1. `gepa-research-version-check: command not found` -- the host's plugin install is incomplete (missing the bin/ wrapper). Fall back to running gepa-research --version directly and check for gepa-research-cli in the output; if it's a different package, tell the user to uninstall it and install gepa-research-cli from GitHub in its place (uv tool install "git+https://github.com/CyrusNuevoDia/gepa-research#subdirectory=plugins/gepa-research").

Do not try to auto-install. Host sandbox + network policy may block it; leaving the install as a user action keeps failure modes clear.

Guiding principles

• Main stays clean. Never commit gepa-research-specific artifacts (benchmark harness, instrumentation, SDK imports) to main. Main should contain only what existed before gepa-research plus anything the user already had. All gepa-research-specific work happens inside worktree 0 (the baseline experiment).
• Baseline is a worktree, not a main commit. gepa-research init creates .gepa-research/ but nothing in main changes. The first real experiment (exp_0000, created by gepa-research new --parent root) is where the benchmark and instrumentation live.
• Ask the user as little as possible. Every question is a beat of friction. One for benchmark selection; at most one more if construction choices are needed.
• Relay the dashboard URL verbatim when it prints. This is the user's window into the run.

1. Explore the repo

Understand what the codebase does. Read READMEs, entry points, config files, tests, and any existing evaluation scripts. Identify:

• The optimization target: which file(s) benefit from iterative optimization?
• Metric direction for each candidate: is higher better (max) or lower better (min)?
• Critical behaviors worth gating: invariants that must never break regardless of score (e.g., "refund flow works", "core tests pass", "output is valid JSON"). Gates are commands that exit 0 on success, non-zero on failure.

2. Look for the obvious benchmark

Check what's already there:

• Full benchmarks: existing scripts that run end-to-end and output a score
• Partial evals: tests, notebooks, or logs with ground truth but not in runnable-score form
• Nothing at all

Also check what the user asked for in the invocation argument. If they named a specific metric or target, that's intent.

If one benchmark is obviously the right one — a runnable eval that measures what the user clearly cares about, or what the repo is plainly built to do — use it. Skip step 3, go to step 4 with that benchmark as the only candidate.

If it's not obvious — multiple candidate surfaces, no existing eval, user didn't specify intent, or the existing eval covers a narrow slice while the interesting optimization sits elsewhere — run step 3.

3. Propose unexplored optimization dimensions (only if step 2 was ambiguous)

When the benchmark isn't obvious, propose candidate dimensions grounded in actual repo signals, then pick with the user. See references/proposing-dimensions.md for the full rubric, project-type examples, and presentation format. Short version:

• A handful of dimensions relevant to this specific repo (not generic categories).
• Ground each in repo signals: already-instrumented code, stated goals in READMEs, TODO/FIXME patterns, domain defaults.
• Rank by signal × slack × cost answered in prose (no numeric scores — they're vibes).

4. Ask the user to pick the benchmark

If step 2 produced one obvious benchmark, confirm it in one sentence and move on — no ranked list needed.

Otherwise, ask once:

"I'm proposing these optimization targets for this repo:

[ranked list with one-line explanations, construction complexity, and whether an existing eval covers some of it]

Which should we optimize? Recommended: [default pick with reasoning]."

Record the selection. If step 3 ran, save non-picked dimensions to .gepa-research/project.md under "Future experiment candidates" after init.

5. Ask the user for instrumentation mode

Three cases, in order of how to handle them:

1. Selected benchmark already exists AND is already instrumented for gepa-research (you can see from gepa_research_agent import Run, an import { Run } from 'gepa-research', or the inline log_task / logTask helpers in the benchmark source). No wiring needed. Skip this question entirely. Detect the instrumentation style from the source and pass the matching --instrumentation-mode <sdk|inline> value to gepa-research init in step 7.

1. Selected benchmark already exists but is NOT instrumented (it just prints a score JSON, or it's a test runner that doesn't yet write per-task traces). Wiring is needed. Ask the question.

1. Selected benchmark needs to be constructed from scratch (case B or C from step 4). Wiring is needed. Ask the question.

For cases 2 and 3, ask once:

"I can wire up the benchmark in one of two ways:

1. SDK mode -- install the SDK directly from this GitHub repo. Python: pip install "git+https://github.com/CyrusNuevoDia/gepa-research#subdirectory=sdk/python". Node: git clone https://github.com/CyrusNuevoDia/gepa-research /tmp/gepa-research && npm install /tmp/gepa-research/sdk/node (npm has no native subdirectory-from-git install, so clone + local-path install is the canonical recipe). Richer per-task logs, ~5 lines of user code.

2. Inline mode -- paste a ~30-line helper directly into the benchmark. Zero new dependencies. Same data contract."

Pass the answer to gepa-research init via --instrumentation-mode <sdk|inline> in step 7. Never install packages without this confirmation. If you skip the question (case 1), still pass the detected mode to gepa-research init so optimize/subagent runs see a consistent value.

6. Prepare main (without committing to it)

The agent never creates commits on main. Main stays byte-identical to what the user committed before gepa-research ran. Two things to set up, both local-only.

Order matters: do 6a (audit) before 6b (excludes). The excludes in 6b will hide files inside node_modules/, dist/, build/, etc. from git status. If you run the audit after adding excludes, you'll be blind to anything missing inside those directories -- and benchmark dependencies often live exactly there.

6a. Detect (don't auto-commit) dirty or untracked dependencies

gepa-research new forks a worktree from the current branch's HEAD commit, not from your dirty working tree. Any uncommitted edits to the target, benchmark, or gate dependencies are silently absent from exp_0000, and the whole optimization tree gets built against stale code while you think gepa-research is running on what you see locally.

Run three checks, in this order:

1. Tracked-but-modified files -- run git diff --name-only and git diff --cached --name-only. If any output line is the optimization target, an existing benchmark file, a gate-referenced script, or any of their import-graph dependencies, stop and ask the user to commit or stash before continuing. Do not commit on their behalf -- the user might be in the middle of an unrelated change.

1. Untracked files visible to git -- run git status --short --untracked-files=all and look for ?? entries that the target or gates will reference. Classify each:

• Part of the user's project (e.g., a smoke test they wrote but hadn't committed) -- stop and ask the user to commit it to main themselves.
• GEPAResearch-specific new files (a new gate script you're about to write, a new test fixture) -- do not create these in main. Defer to step 10; they go into the baseline worktree and commit to experiment 0's branch. Every descendant experiment inherits via git branching.

1. Explicit paths inside soon-to-be-ignored directories -- inspect the benchmark command and every gate command for path references (e.g., ./dist/eval-helper, node_modules/some-tool/cli.js, build/golden_outputs/). For each such path, run git ls-files --error-unmatch <path> to confirm it's tracked. If any aren't, stop and ask the user to commit them. This catches dependencies that step 6b is about to hide from git status.

Any one of these three checks failing is a hard stop. Do not proceed to 6b or beyond until the working tree is clean with respect to anything gepa-research will read.

Anything else (benchmark harness, instrumentation) always gets constructed inside the baseline worktree, never in main.

6b. Add local-only git excludes

After the audit passes, append to .git/info/exclude (not .gitignore -- we do not commit to main):

.gepa-research/
__pycache__/
*.pyc
.pytest_cache/
node_modules/
dist/
build/

.git/info/exclude is git's per-clone ignore file -- same effect as .gitignore, but never committed, never shared, invisible to history. Right tool for per-machine tooling state.

7. Initialize the workspace

gepa-research init --target <file> --benchmark "<command using {worktree} and {target}>" --metric <max|min> \
  --instrumentation-mode <sdk|inline> [--gate "<gate command>"] \
  [--objective "<one-sentence optimization goal>"] \
  [--background "<domain knowledge / constraints for the LLM>"] \
  [--reflection-lm "<provider/model, e.g. anthropic/claude-opus-4-7>"]

The last three flags are stored in config.json and forwarded to gepa.optimize_anything during /optimize. They are optional but strongly recommended — GEPA's reflection prompts are materially better with a clear objective than with a generic "improve the score".

Good --objective values are one sentence naming the behavior, not the artifact. "Maximize the pass rate on the tau3 benchmark while keeping latency under 200ms per task." is better than "Improve solve.py".