heshamfs/materials-simulation-skills/simulation-validator

Simulation Validator

Goal

Provide a three-stage validation protocol: pre-flight checks, runtime monitoring, and post-flight validation for materials simulations.

Requirements

• Python 3.10+
• No external dependencies (uses Python standard library only)
• Works on Linux, macOS, and Windows

Inputs to Gather

Before running validation scripts, collect from the user:

Decision Guidance

When to Run Each Stage

Is simulation about to start?
├── YES → Run Stage 1: preflight_checker.py
│         └── BLOCK status? → Fix issues, do NOT run simulation
│         └── WARN status? → Review warnings, document if accepted
│         └── PASS status? → Proceed to run simulation
│
Is simulation running?
├── YES → Run Stage 2: runtime_monitor.py (periodically)
│         └── Alerts? → Consider stopping, check parameters
│
Has simulation finished?
├── YES → Run Stage 3: result_validator.py
│         └── Failed checks? → Do NOT use results
│                            → Run failure_diagnoser.py
│         └── All passed? → Selected metric checks passed; physical validity still needs model-specific evidence

Choosing Validation Thresholds

Script Outputs (JSON Fields)

Three-Stage Validation Protocol

Stage 1: Pre-flight (Before Simulation)

1. Run scripts/preflight_checker.py --config simulation.json
2. BLOCK status: Stop immediately, fix all blocker issues
3. WARN status: Review warnings, document accepted risks
4. PASS status: Proceed to simulation

Note: preflight_checker.py validates required keys, numeric ranges,

output-directory access, and disk space. It does not evaluate numerical

stability (CFL / diffusion-Fourier). For explicit stability gating use

skills/core-numerical/numerical-stability/scripts/cfl_checker.py.

python3 scripts/preflight_checker.py \
    --config simulation.json \
    --required dt,dx,kappa \
    --ranges "dt:1e-6:1e-2,dx:1e-4:1e-1" \
    --min-free-gb 1.0 \
    --json

Stage 2: Runtime (During Simulation)

1. Run scripts/runtime_monitor.py --log simulation.log periodically
2. Configure alert thresholds based on problem type
3. Stop simulation if critical alerts appear

python3 scripts/runtime_monitor.py \
    --log simulation.log \
    --residual-growth 10.0 \
    --dt-drop 100.0 \
    --json

Stage 3: Post-flight (After Simulation)

1. Run scripts/result_validator.py --metrics results.json
2. All checks PASS: Results are valid for analysis
3. Any check FAIL: Do NOT use results, diagnose failure

python3 scripts/result_validator.py \
    --metrics results.json \
    --bound-min 0.0 \
    --bound-max 1.0 \
    --mass-tol 1e-3 \
    --json

For variational / gradient-flow models (Allen-Cahn, Cahn-Hilliard), add --variational to enforce a strict monotone non-increasing energy check.

Failure Diagnosis

When validation fails:

python3 scripts/failure_diagnoser.py --log simulation.log --json

Conversational Workflow Example

User: My phase field simulation crashed after 1000 steps. Can you help me figure out why?

Agent workflow:

1. First, check the log for obvious errors:

``bash python3 scripts/failure_diagnoser.py --log simulation.log --json ``

1. If diagnosis suggests numerical blow-up, check runtime stats:

``bash python3 scripts/runtime_monitor.py --log simulation.log --json ``

1. Recommend fixes based on findings:

• If residual grew rapidly → reduce time step
• If dt collapsed → check stability conditions
• If NaN detected → check initial conditions

Error Handling

Interpretation Guidance

Status Meanings

Confidence Score Interpretation

A requested bound (--bound-min/--bound-max) with no matching field_min/field_max in the metrics is reported as a failed bounds_unverifiable check, never a vacuous pass. For variational/gradient-flow runs, pass --variational (or set "energy_variational": true in the metrics) to enforce a strict monotone non-increasing energy check (energy_monotone); otherwise a weaker energy_net_decrease check is used, which does not detect mid-run spikes.

Common Failure Patterns

Verification checklist

Do not trust a validation verdict until each applicable item below is satisfied with the concrete artifact named. Record these in your summary to the user.

• [ ] Ran result_validator.py --json and confirmed results.status is PASS (not INSUFFICIENT_DATA) AND results.confidence_score == 1.0; a null score or INSUFFICIENT_DATA means no check ran — treat as unverified, not as a pass.
• [ ] Listed results.checks and confirmed every requested check actually appears (e.g. mass_conserved, bounds_satisfied, no_nan, and energy_monotone/energy_net_decrease); confirmed results.failed_checks is empty and contains no bounds_unverifiable entry (which means a requested bound had no field_min/field_max to compare against).
• [ ] For variational/gradient-flow models (Allen-Cahn, Cahn-Hilliard), passed --variational (or set "energy_variational": true) so energy_monotone is enforced; recorded that the weaker energy_net_decrease was NOT relied on, since it cannot detect mid-run energy spikes.
• [ ] Recorded the mass drift tolerance used (--mass-tol, default 1e-3) and confirmed it matches the Conservative/Standard/Relaxed column appropriate to the run; did not silently accept the default for a tight-conservation problem.
• [ ] Ran runtime_monitor.py --json and recorded residual_stats (min/max/last) and dt_stats; confirmed there are no alerts for NaN/Inf/overflow, residual growth above --residual-growth, or dt collapse below --dt-drop.
• [ ] Confirmed numerical stability was gated separately via core-numerical/numerical-stability/scripts/cfl_checker.py (CFL/Fourier limit) — preflight_checker.py does NOT evaluate CFL/Fourier and a PASS preflight says nothing about temporal/spatial stability.
• [ ] On any FAIL or alert, ran failure_diagnoser.py --json and recorded the probable_causes/recommended_fixes, rather than reusing the results.

Common pitfalls & rationalizations

Security