grind-lab-core/night_owl_research_agent/training-check

Skill: training-check

You monitor spatial experiment execution and detect problems early to avoid wasting compute time.

Phase 1: Check Active Experiments

Read output/EXPERIMENT_LOG.md for experiments with status RUNNING or PENDING.

For each running experiment:

1. Check if output file was recently modified:

python -c "import os, time; f='[output_path]'; age=(time.time()-os.path.getmtime(f))/60; print(f'Modified {age:.1f} min ago')"

1. Check last few lines of log file for errors:

tail -20 [log_path]

1. Check for error keywords: Error, Traceback, NaN, inf, MemoryError, Killed
2. Check for stall: no file modification in > 30 min despite RUNNING status

Phase 2: Classify Status

Phase 3: Actions

COMPLETE: Update output/EXPERIMENT_LOG.md status to SUCCESS. Append finding to output/PROJ_NOTES.md. CLEARLY BAD: Kill job if possible. Mark as FAILED in log. Try to fix the issue and retry once. STALLED: Check if process is still alive. If dead: re-queue. If alive: wait 15 more min.

Phase 3.5: Human Checkpoint — Data Synthesis

Honor the HUMAN_CHECKPOINT flag in CLAUDE.md (default: true). This is a monitoring skill, but a few automatic recovery actions silently produce or substitute experiment data. When HUMAN_CHECKPOINT is true, PAUSE and request explicit user approval before any of the following; when false, log the action to output/PROJ_NOTES.md with rationale and proceed.

Do not mark a run SUCCESS in EXPERIMENT_LOG.md based on synthesized metrics. If the user approves, append Synthesis approved: <action> — <user reason> — <date> to the run's notes.

Phase 4: Progress Report

Output to stdout:

Training Check — <timestamp>
Active experiments: N
  COMPLETE: N
  OK (running): N
  STALLED: N
  FAILED: N
 
[List any failures or alerts]

Append alerts to output/PROJ_NOTES.md: [ALERT] Experiment <name> failed: <reason>