grind-lab-core/night_owl_research_agent/deploy-experiment

Deploy Experiment

Execute the experiment described in output/refine-logs/EXPERIMENT_PLAN.md (with context from output/refine-logs/FINAL_PROPOSAL.md) and write all results to output/experiment/.

Scope override (optional): $ARGUMENTS

Inputs

If either required input is missing, stop and tell the user which file to produce first (likely via /refine-research and /experiment-design).

Outputs (all under output/experiment/)

Create the output/experiment/, output/experiment/data/, output/experiment/figures/, and output/experiment/scripts/ directories at the start if they do not exist.

Step 0: Mandatory Local GPU Availability Check (pilot AND full experiments)

This check is REQUIRED before classification, regardless of whether the run is a pilot, a full experiment, Track A, Track B, or mixed. The goal is to guarantee that when a local GPU is present, every ML / DL component runs on it — never silently on CPU.

Run the local GPU presence check first:

Local CUDA:

nvidia-smi --query-gpu=index,name,memory.used,memory.total --format=csv,noheader

Local Mac MPS (only if `nvidia-smi` is missing or returns zero rows):

python -c "import torch; print('MPS available:', torch.backends.mps.is_available())"

Decision rules (apply in order):

1. If nvidia-smi exits 0 and returns at least one GPU row → record LOCAL_GPU=cuda, capture the <gpu_id> and the chosen device for use in Step A5 and any pilot launch. Do NOT gate on idle memory; a desktop GPU with a display attached normally reports 0.7–1.5 GiB used.
2. Else if torch.backends.mps.is_available() returns True → record LOCAL_GPU=mps.
3. Else → record LOCAL_GPU=none. Only in this case may an ML / DL component fall back to CPU or to remote / Modal as defined in CLAUDE.md.

Write the result as the first lines of output/experiment/EXPERIMENT_LOG.md:

-LOCAL_GPU: cuda | mps | none
-Device chosen: <e.g. CUDA:0 / mps / cpu / remote-A100>
-GPU check command + output: <verbatim>

Hard rule: if LOCAL_GPU is cuda or mps, every ML / DL pilot AND every ML / DL full-experiment run launched by this skill MUST use that local GPU. Do not switch to remote / Modal / CPU just because the plan's default is set elsewhere — local GPU takes precedence whenever it is present. Record any deviation as a CONTRACT_VIOLATION.md with the user's explicit override.

If LOCAL_GPU=none and CLAUDE.md lists gpu: remote or gpu: modal, route ML / DL work there as documented in Step A1; otherwise stop and report to the user before proceeding.

This check applies equally to:

• Pilot experiments invoked from /generate-idea (Phase 5)
• Full experiments invoked from /full-pipeline Stage 2
• Any mixed GeoAI experiment whose Track A leg requires GPU

Step 0.1: Classify the Experiment

Read EXPERIMENT_PLAN.md and FINAL_PROPOSAL.md, then classify into ONE of the two tracks below. Record the choice at the top of EXPERIMENT_LOG.md.

Mixed experiments (e.g., GeoAI: deep learning applied to spatial data) run both tracks in sequence — Track A for model training/inference, Track B for downstream spatial evaluation.

Invoke supporting skills when needed:

• If the plan requires datasets not yet in data/raw/ → call `data-download` skill with the data requirements from the plan.
• For Track B (spatial analysis execution and diagnostics) → call `spatial-analysis` skill to run the analysis under guideline-driven decisions. This skill (deploy-experiment) orchestrates and records results; spatial-analysis is the executor for spatial methods.

Track A — ML / Deep Learning Experiments

A1. Detect Environment

From CLAUDE.md:

• Local GPU (gpu: local): local CUDA or Apple MPS
• Remote server (gpu: remote): SSH alias, conda env, code directory
• Modal (gpu: modal): use Modal serverless GPU app

A2. Pre-flight GPU Check

The local presence check from Step 0 has already determined LOCAL_GPU. Re-use that result — do not re-decide here. Only run remote / multi-GPU contention checks below.

Remote (SSH), only when `LOCAL_GPU=none` and `CLAUDE.md` says `gpu: remote`:

ssh <server> nvidia-smi --query-gpu=index,memory.used,memory.total --format=csv,noheader

Contention check (optional, only when sharing a multi-GPU box): pick a GPU where memory.free / memory.total > 0.7 to avoid colliding with another running job. On a single-GPU workstation, skip this and proceed once presence is confirmed in Step 0.

A3. Sync Code (Remote Only)

Read code_sync in CLAUDE.md (default rsync).

rsync:

rsync -avz --include='*.py' --exclude='*' <local_src>/ <server>:<remote_dst>/

git:

git add -A && git commit -m "sync: experiment deployment" && git push
ssh <server> "cd <remote_dst> && git pull"

A4. W&B Integration (only when wandb: true)

Skip entirely if not enabled. Otherwise, ensure scripts contain:

import wandb
wandb.init(project=WANDB_PROJECT, name=EXP_NAME, config={...})
wandb.log({"train/loss": loss, "train/lr": lr, "step": step})
wandb.log({"eval/loss": eval_loss, "eval/accuracy": acc})
wandb.finish()

Verify login: ssh <server> "wandb status".

A5. Launch

Remote (SSH + screen):

ssh <server> "screen -dmS <exp_name> bash -c '\
  eval \"\$(<conda_path>/conda shell.bash hook)\" && \
  conda activate <env> && \
  CUDA_VISIBLE_DEVICES=<gpu_id> python <script> <args> 2>&1 | tee <log_file>'"

Local:

CUDA_VISIBLE_DEVICES=<gpu_id> python <script> <args> 2>&1 | tee output/experiment/<exp_name>.log
# Mac MPS: omit CUDA_VISIBLE_DEVICES

Use run_in_background: true for long local jobs. Each experiment gets its own screen/process and one GPU.

A6. Collect Artifacts

After completion (detected via /monitor-experiment or screen ending):

rsync -avz -e "ssh -p <PORT>" root@<HOST>:/workspace/project/results/ output/experiment/data/
scp -P <PORT> root@<HOST>:/workspace/*.log output/experiment/

Place tabular metrics as metrics.csv / metrics.json, learned checkpoints' evaluation predictions as predictions.parquet (or .npz), and any training curves as curves.csv — all under output/experiment/data/ so they are visualization-ready.

Track B — Spatial / GIScience Experiments

These experiments are typically CPU-bound, run locally, and rely on PyGeoAI / GeoPandas / PySAL / libpysal / esda / spreg / mgwr / rasterio / xarray / networkx / scikit-mobility. GPU is irrelevant unless the plan specifies a deep-learning component (then also run Track A).

B1. Verify / Acquire Data

1. Read the Data section of EXPERIMENT_PLAN.md.
2. Check data/DATA_MANIFEST.md and data/raw/ for each required dataset.
3. For any missing dataset → invoke the `data-download` skill with the precise data need (geography, time range, spatial resolution, variables, format). Do NOT ad-hoc download; rely on the skill so provenance is recorded in DATA_MANIFEST.md.
4. After download, re-check presence and basic integrity (file exists, non-zero size, opens with the expected library).

B2. Pre-flight Environment Check

python - <<'PY'
import importlib, sys
required = ["geopandas","pysal","libpysal","esda","spreg","mgwr","rasterio","xarray","shapely","pyproj","numpy","pandas","matplotlib"]
missing = [p for p in required if importlib.util.find_spec(p) is None]
print("MISSING:", missing)
PY

If anything is missing, report to the user and stop — do NOT silently pip install into a shared env. Suggest the conda env from CLAUDE.md.

B3. Plan-to-Run Mapping

For every claim listed in EXPERIMENT_PLAN.md, identify:

B4. Execute via the spatial-analysis Skill

Invoke the spatial-analysis skill per claim (or per grouped set of claims). Pass the research question verbatim from EXPERIMENT_PLAN.md and the resolved data paths. The skill will write analysis artifacts to output/spatial-analysis/ — this skill (deploy-experiment) is then responsible for:

1. Copying / linking the resulting tables and figures into output/experiment/data/ and output/experiment/figures/ with stable filenames.
2. Writing a short per-claim summary row (numbers, CI, p-values, effect sizes, diagnostics) to EXPERIMENT_RESULT.md.

Intermediate data files to persist — decide which artifacts are actually needed based on the claims, methods, and visualizations the plan implies. The list below is illustrative, not mandatory; produce only what is genuinely useful for downstream visualization, paper figures, or reproducibility, and skip the rest.

Examples (pick, rename, or add as appropriate):

• data/features.gpkg or .parquet — joined analytical table with geometries
• data/moran_results.csv — global + local indicators
• data/lisa_clusters.gpkg — LISA cluster/outlier labels
• data/regression_coefficients.csv — OLS / spatial lag / error / GWR coefficients
• data/gwr_local_estimates.gpkg — per-location coefficients, t-values, local R²
• data/predictions.parquet — out-of-sample predictions with geometry keys
• data/diagnostics.json — VIF, condition number, Moran's I on residuals, CV scores
• data/metrics.json — aggregate pass/fail per claim (machine-readable)

Always accompany data with metadata so the next step (figure generation, paper writing, review) can interpret it without re-reading code: