heshamfs/materials-simulation-skills/simulation-orchestrator

Simulation Orchestrator

Goal

Provide tools to manage multi-simulation campaigns: generate parameter sweeps, track job execution status, and aggregate results from completed runs.

Requirements

• Python 3.10+
• No external dependencies (uses Python standard library only)
• Works on Linux, macOS, and Windows

Inputs to Gather

Before running orchestration scripts, collect from the user:

Decision Guidance

Choosing a Sweep Method

Need every combination (full factorial)?
├── YES → Use grid (warning: exponential growth with parameters)
└── NO → Is space-filling coverage needed?
    ├── YES → Use lhs (Latin Hypercube Sampling)
    └── NO → Use linspace for uniform sampling per parameter

Campaign Size Guidelines

Script Outputs (JSON Fields)

Note on swept parameter names: sweep_generator.py writes each swept value into the base config by key path. A bare name (e.g. kappa) overwrites a top-level key; a dot-notation name (e.g. parameters.kappa) targets a nested key. The swept key path must match where the solver reads the value — sweeping kappa against a config that nests parameters.kappa would add an unused top-level key and silently leave the base value in place. See references/sweep_strategies.md.

Workflow

Step 1: Generate Parameter Sweep

Create configurations for all parameter combinations:

python3 scripts/sweep_generator.py \
    --base-config base_config.json \
    --params "dt:1e-4:1e-2:5,kappa:0.1:1.0:3" \
    --method linspace \
    --output-dir ./campaign_001 \
    --json

Step 2: Initialize Campaign

Create campaign tracking structure:

python3 scripts/campaign_manager.py \
    --action init \
    --config-dir ./campaign_001 \
    --command "python sim.py --config {config}" \
    --json

Step 3: Track Job Status

Monitor running jobs:

python3 scripts/job_tracker.py \
    --campaign-dir ./campaign_001 \
    --update \
    --json

Step 4: Aggregate Results

Combine results from completed runs:

python3 scripts/result_aggregator.py \
    --campaign-dir ./campaign_001 \
    --metric final_energy \
    --json

result_aggregator.py minimizes by default: best_run is the run with the lowest metric value (and summary.minimize is true). If higher is better (e.g. yield, accuracy, throughput), pass --maximize so best_run becomes the highest value:

# Higher is better -> select the maximum
python3 scripts/result_aggregator.py \
    --campaign-dir ./campaign_001 \
    --metric yield \
    --maximize \
    --json

Decision guidance: If higher is better (yield, accuracy, throughput), pass

--maximize; otherwise the reported best_run is the minimum.

CLI Examples

# Generate 5x3=15 runs varying dt (5 values) and kappa (3 values)
python3 scripts/sweep_generator.py \
    --base-config sim.json \
    --params "dt:1e-4:1e-2:5,kappa:0.1:1.0:3" \
    --method linspace \
    --output-dir ./sweep_001 \
    --json
 
# Generate LHS samples for 4 parameters with budget of 20 runs
python3 scripts/sweep_generator.py \
    --base-config sim.json \
    --params "dt:1e-4:1e-2,kappa:0.1:1.0,M:1e-6:1e-4,W:0.5:2.0" \
    --method lhs \
    --samples 20 \
    --output-dir ./lhs_001 \
    --json
 
# Check campaign status
python3 scripts/campaign_manager.py \
    --action status \
    --config-dir ./sweep_001 \
    --json
 
# List jobs (read-only), optionally filtered by status
python3 scripts/campaign_manager.py \
    --action list \
    --config-dir ./sweep_001 \
    --status-filter failed \
    --json
 
# Get summary statistics from completed runs (minimize: best = lowest)
python3 scripts/result_aggregator.py \
    --campaign-dir ./sweep_001 \
    --metric final_energy \
    --json
 
# Maximization metric: best = highest value (yield, accuracy, throughput)
python3 scripts/result_aggregator.py \
    --campaign-dir ./sweep_001 \
    --metric yield \
    --maximize \
    --json

Conversational Workflow Example

User: I want to run a parameter sweep on dt and kappa for my phase-field simulation. I want to try 5 values of dt between 1e-4 and 1e-2, and 4 values of kappa between 0.1 and 1.0.

Agent workflow:

1. Calculate total runs: 5 x 4 = 20 runs
2. Generate sweep configurations:

``bash python3 scripts/sweep_generator.py \ --base-config simulation.json \ --params "dt:1e-4:1e-2:5,kappa:0.1:1.0:4" \ --method linspace \ --output-dir ./dt_kappa_sweep \ --json ``

1. Initialize campaign:

``bash python3 scripts/campaign_manager.py \ --action init \ --config-dir ./dt_kappa_sweep \ --command "python phase_field.py --config {config}" \ --json ``

1. After user runs simulations, aggregate results:

``bash python3 scripts/result_aggregator.py \ --campaign-dir ./dt_kappa_sweep \ --metric interface_width \ --json ``

Error Handling

Integration with Other Skills

The simulation-orchestrator works with other simulation-workflow skills:

parameter-optimization          simulation-orchestrator
        │                              │
        │ DOE samples ────────────────>│ Generate configs
        │                              │
        │                              │ Run simulations
        │                              │
        │<──────────────────────────── │ Aggregate results
        │                              │
        │ Sensitivity analysis         │
        │ Optimizer selection          │

Typical Combined Workflow

1. Use parameter-optimization/doe_generator.py to get sample points
2. Use simulation-orchestrator/sweep_generator.py to create configs
3. Run simulations (user's responsibility)
4. Use simulation-orchestrator/result_aggregator.py to collect results
5. Use parameter-optimization/sensitivity_summary.py to analyze

Verification checklist

Before trusting a campaign's best_run or summary statistics, record concrete evidence for each item:

• [ ] Confirmed the swept key path actually changed the value the solver reads: opened at least one generated config_NNNN.json and verified the swept parameter (e.g. parameters.kappa) holds the expected value at the expected nesting level, not a duplicate unused top-level key (sweep_generator.py writes by key path).
• [ ] Reconciled job accounting from result_aggregator.py --json: recorded summary.total_jobs, summary.completed, and summary.failed, and confirmed completed + failed == total_jobs. Any shortfall means runs were silently skipped (missing result file or extract_metric returned None) and must be investigated, not ignored.
• [ ] Confirmed completed > 0 and that the recorded summary.metric matches the field the solver actually writes. A typo'd or absent metric makes extract_metric return None, yielding zero completed runs with no error.
• [ ] Recorded summary.minimize and confirmed it matches the intended direction (default minimize; --maximize for yield/accuracy/throughput) before quoting best_run.
• [ ] Did NOT treat job_tracker.py "completed" as physical success: it flags a job completed purely from a result-file's existence and stamps exit_code 0 — independently checked the run's real exit status / solver logs for non-zero codes or NaN/Inf output.
• [ ] Applied an outlier/sanity check to the metric values (e.g. Tukey 1.5x IQR from references/aggregation_methods.md) and confirmed best_run.value is physically plausible, not a crashed run that emitted a spurious extremum.
• [ ] For LHS sweeps, recorded the --seed used and saved manifest.json (parameter bounds, total_runs, parameter_space) so the sample set is reproducible.

Common pitfalls & rationalizations