csml-rpi/ai-cfd-scientist/cfd-open-discovery

cfd-open-discovery

⚠ HARD STOP — read this before any other step

If <out-dir>/checkpoints/mesh_gate_done.json does not exist, STOP. The orchestrator chain enters this skill only after literature, baseline_setup, metric_setup, and mesh-gate have completed. The discovery loop's compile-run-score iteration requires a locked mesh and a baseline metric set — running it without them produces unscored candidates against a moving target. Return to Skill cfd-orchestrator and walk the chain from literature.

Hard preflight — the mesh-gate checkpoint must be real, not fabricated. Run this check as Step 0 of this skill, before any iteration:

OUT=<out-dir>
CP=$OUT/checkpoints/mesh_gate_done.json
test -f $CP || { echo "STOP: mesh_gate_done.json missing — invoke Skill cfd-skills/cfd-mesh-gate first"; exit 1; }
HAS_SPEC=$(test -f $OUT/selected_mesh_spec.json && echo 1 || echo 0)
HAS_CTX=$(test -f $OUT/mesh_independence_context.json && echo 1 || echo 0)
HAS_DIR=$(test -d $OUT/mesh_gate && echo 1 || echo 0)
STARTER_LOCKED=$(jq -r 'select((.status // "") == "starter_mesh_locked" or .starter_mesh_locked == true) | "1"' $CP 2>/dev/null)
if [ "$HAS_SPEC$HAS_CTX$HAS_DIR" = "000" ] && [ "$STARTER_LOCKED" != "1" ]; then
  echo "STOP: mesh_gate_done.json exists but no mesh-gate artifacts on disk"
  echo "  (no selected_mesh_spec.json, no mesh_independence_context.json, no mesh_gate/ dir)"
  echo "  AND the checkpoint does NOT declare status='starter_mesh_locked'."
  echo "  The checkpoint was fabricated to bypass the mesh-gate stage."
  echo ""
  echo "  Two valid recoveries:"
  echo "  (a) Invoke Skill cfd-skills/cfd-mesh-gate properly, run the baseline + refined"
  echo "      comparison, and write selected_mesh_spec.json + mesh_independence_context.json."
  echo "  (b) If the starter ships a mesh whose grid-independence is established in the"
  echo "      peer-reviewed literature, re-emit mesh_gate_done.json with a VERIFIABLE"
  echo "      citation (see cfd-mesh-gate 'Starter-mesh lock'):"
  echo "        {\"stage\": \"mesh_gate\","
  echo "         \"status\": \"starter_mesh_locked\","
  echo "         \"mesh_source\": \"<path to starter polyMesh / blockMeshDict>\","
  echo "         \"citation\": {\"doi\": \"10.xxxx/xxxxx\", \"title\": \"...\","
  echo "                       \"what_it_validates\": \"...\"},"
  echo "         \"rationale\": \"<why locking is correct for this study>\"}"
  echo "      The audit H31 gate resolves the DOI live; a free-text rationale alone fails."
  exit 1
fi
echo "OED preflight OK: mesh-gate backed by artifacts (spec=$HAS_SPEC ctx=$HAS_CTX dir=$HAS_DIR starter_locked=${STARTER_LOCKED:-0})"

The audit's H31 gate enforces the same rule post-hoc. The preflight above catches the fabrication before OED burns 30+ iterations of compute on a moving mesh.

Iterative model discovery: propose candidate → compile → run → score → accept or revise → continue until --open-ended-budget is exhausted or improvement threshold is met.

The loop is fully self-contained. Multi-metric tracking + LLM-as-judge are always on (the loop reads the topic + reference data + baseline postProcessing at startup and decides what to track).

When to use

Topics with phrases: "open-ended discovery", "find a novel", "discover a model", "beat baseline", "best model for", "propose new terms".

Inputs

• out-dir (required)
• topic (required) — natural-language goal (e.g. "novel SA modification for periodic hill Re=5600 beating baseline on Cf")
• starter-dir (required) — baseline case + reference data (e.g. DNS Cf)
• budget (required) — max iterations (e.g. 20)
• baseline_case (optional) — if not specified, the first iteration runs the unmodified baseline
• improvement_threshold (optional, default 5%) — required improvement vs baseline to accept
• diversity_mode (optional, off | hybrid | aggressive, default off)
• multi_flow_starter_dirs (optional) — list of additional flow folders for multi-flow validation

Starter directory(ies) are READ-ONLY (per skills/cfd-orchestrator/SKILL.md non-negotiable constraints). Reference data CSVs/PDFs and base-case files are inputs only — every modification (custom-model build, candidate case run, comparator authoring) lives under <out-dir>/open_ended_discovery/iter_<NNN>_*/. Per-iter dirs cp -r from the starter; never touch the starter tree.

Outputs

• <out-dir>/open_ended_discovery/history.json — iteration log
• <out-dir>/open_ended_discovery/candidates/<id>/ — per-candidate code + run output
• <out-dir>/open_ended_discovery/best.json — best model found, params, score
• <out-dir>/baseline_metrics.json — baseline scores
• <out-dir>/oed_artifact.json — handoff descriptor (see schema below)

Agentic OED loop discipline (skill mode)

The setup + discovery + termination steps below tell you what each iteration does. This section tells you how to drive the iterative loop as the agent runtime — without handing off to scripts/open_ended_discovery.py. It's the skill-mode counterpart to that 4.7k-line Python supervisor.

The four-action inner loop

Each iteration, you pick exactly one of four actions and pay its budget cost:

Persist budget state in <out-dir>/open_ended_discovery/ext_state.json#budget_used — write it after every iteration.

Per-iteration library packaging (item 11 — no cross-iteration dependencies)

Each iteration's case must be self-contained. cfd-code-modify builds the custom library case-local at <iter case>/customModels/<Class>/lib<Class>.so, and controlDict.libs references it by that case-relative path.

When an experiment action reuses a library an earlier code_mod iteration compiled, copy the whole customModels/<Class>/ directory (with the compiled .so) into this iteration's case — do not symlink to, or load by absolute path from, the earlier iteration's directory. A dependency like iter_118/.../libX.so → iter_060/.../libX.so breaks the moment iter_060 is cleaned or the bundle is moved, and is exactly the cross-iteration runtime-library defect the audit now rejects.

Every iteration's case therefore carries each .so it loads inside its own customModels/. The runtime_dependencies.json check (cfd-code-modify Step 7b) confirms there is no cross-iteration dependency.

Stop conditions (any one fires → write oed_artifact.json and stop)

1. Budget exhausted: budget_used >= budget_total.
2. No-progress cap: 3 consecutive iterations with result ∈ {INVALID, REVISE, FAILED}. Without this cap, an OED loop with a broken measurement layer (extractor reading the wrong field; comparator returning a constant) will burn its entire budget re-discovering the same null result instead of surfacing the underlying gap. Write stop_reason: "no_progress_streak" to the artifact and surface the failure mode in oed_artifact.json.
3. Convergence: the last 3 accepted candidates show improvement < 0.5% over the running best. Diminishing returns; no point spending more budget.
4. Manual: <out-dir>/open_ended_discovery/STOP file exists (touch it to halt mid-run cleanly).

Verifier-independence rule (critical)

The general failure mode this rule prevents: a comparator and a verifier that both go through the same data-access library on the same field will agree with each other even when both are reading the wrong thing. Self-consistent agreement on a shared blind spot is not validation. This is independent of topic — it has been observed for droplet-evaporation, source-term, and reaction-rate measurements alike, anywhere an extractor can hit a constant-by-design or sparsely-written field and not realize.

Discipline:

• If the comparator authored a PyVista extractor, the verifier must use a non-PyVista path (log-parsing, postProcessing/<functionObject>/<time>/<file> table reads, or integrated species/state-variable mass).
• If the comparator authored a text/log extractor, the verifier must use PyVista on a different field.
• Use cfd-skills/cfd-stats/SKILL.md Pattern 7 (two-extractor agreement check) — the relative disagreement must be < 50% for verifier_verdict: OK.
• Without an independent path, the comparator's selftest is provisional only; flag it verifier_verdict: PROVISIONAL and don't trust ranking on it alone.

Per-QoI extraction guidance

The comparator-author LLM doesn't have intrinsic physics knowledge that some OpenFOAM fields are unreliable for cumulative measurements. Splice the appropriate guidance from cfd-skills/cfd-stats/SKILL.md into the comparator-author user prompt:

If the topic mentions a QoI not in this table, add an entry inline in the comparator-author user prompt, then update cfd-skills/cfd-stats/SKILL.md afterwards so the next study benefits.

Iteration directory layout (anti-collision)

Each iteration gets its own dir:

<out-dir>/open_ended_discovery/iter_<NNN>_<action>_<short_label>/

Where <NNN> is a strictly numeric, zero-padded 3-digit integer (001, 002, … 140) that equals the integer `iter` id of the history record. Naming is enforced — the audit's H33 gate fails on irregular ids:

• No letter suffixes (iter_04x), no compound tokens (iter_05R10, iter_0507), no b/x variants (iter_040b).
• Each integer is used exactly once — never reuse a number for a retry; allocate the next one.
• <action> is python_script | code_mod | experiment; <short_label> is a short intent hint.

Per-iter artifacts that must exist before incrementing the iteration counter:

• agentic_result.json (for code_mod / experiment) — status, error, compiled_so / case_dir
• agentic_trajectory.log — what the agent did this iter (file edits, bash commands, durations)
• score.json — a JSON file (never a directory) holding this iteration's metric vector + primary score. When a comparator is run with --out <dir>, that output directory must not be named score.json; use _diag/ or similar. The audit's H33 gate fails any score.json path that is a directory.
• For python_script: analysis_script.py + output.txt

Hard per-iteration history gate (item 12). <out-dir>/open_ended_discovery/history.json must gain exactly one record the moment an iteration directory is created — before the next iteration starts. Check it every iteration: