kwongfuk/codex-skills/palmetto-slurm-safe-guard

Palmetto Slurm Safe Guard

Use This Skill

Use this every time you write or edit a Slurm job script. This is mandatory for long-running jobs (translation/training/inference/eval). When the job is a training job, also apply exact-training-resume-guard. Use palmetto-slurm-workflow alongside this skill for allocation/submission workflow details.

Required Guards (Must Include)

• Safety validation before heavy work:
• reject login node execution
• check GPU tools (nvidia-smi)
• check required GPU count
• validate input file existence
• /scratch storage policy:
• env/cache/model/data/container paths must be under /scratch/$USER
• if the job is launched from a Codex/agent session, verify that $USER is the real Palmetto account before using it in scratch/home paths; if not, set an explicit SCRATCH_ROOT=/scratch/<real_user> and use that consistently
• training checkpoints and any checkpoint-containing output_dir must live under /scratch/$USER (or /scratch/<real_user> in agent contexts), not /home
• large outputs, large logs, raw predictions, intermediate artifacts, and resumable run directories must live under /scratch/$USER
• keep lightweight code-generated logs, manifests, compact metrics, and standard small experiment results inside the project directory unless the user explicitly requests another location
• if a framework mixes checkpoints with TensorBoard/event files inside one output_dir, put that output_dir on scratch and keep separate run logs under project-local logs/
• Model cache and offline policy:
• pre-cache every model/tokenizer/processor repo used by the job under /scratch/$USER/.hf_cache before starting the main workload
• run production training/eval/inference jobs in offline mode by default: HF_HUB_OFFLINE=1, TRANSFORMERS_OFFLINE=1, HF_DATASETS_OFFLINE=1
• if offline mode fails due to missing files, fail fast and report cache preload incomplete instead of downloading during the run
• Checkpointed result saving:
• periodic saves by rows/batches/time
• atomic file write (.tmp then replace)
• persist progress state JSON
• Exact training resume for training jobs:
• save full training state, not model-only checkpoints
• preserve optimizer / scheduler / RNG / framework state needed for exact resume
• auto-detect the latest complete checkpoint and resume from it
• skip incomplete checkpoints instead of resuming from partial files
• Signal-safe termination:
• handle SIGTERM / SIGINT
• force final save before exit
• Runtime monitoring without log spam:
• background monitor loop every MONITOR_INTERVAL seconds (default 120)
• emit concise GPU util/memory and key process lines
• kill monitor in trap ... EXIT

Slurm Template Checklist

1. Preflight

• set -euo pipefail
• node class check (login vs compute)
• nvidia-smi exists and GPU count meets requirement
• input/output paths validated

1. Scratch Paths

• HF_HOME=/scratch/$USER/.hf_cache
• PIP_CACHE_DIR=/scratch/$USER/.pip_cache
• env under /scratch/$USER/envs/...
• optional containers under /scratch/$USER/containers
• datasets, model weights, checkpoints, and large result directories under /scratch/$USER/...
• for Codex/agent-submitted jobs, prefer SCRATCH_ROOT=/scratch/<real_user> over raw $USER expansion if $USER may resolve to coder
• checkpoint trees and checkpoint-writing output_dir under /scratch/$USER/...
• repository keeps code, configs, manifests, lightweight logs, lightweight metrics, and standard small experiment results
• if artifacts are large, append-heavy, or likely to exceed home quota, place them on scratch and keep only small summaries or pointers in the repository
• pre-warm model repos into HF_HOME before the main Python command
• set offline defaults in job scripts: HF_HUB_OFFLINE=1, TRANSFORMERS_OFFLINE=1, HF_DATASETS_OFFLINE=1

1. Logging and Monitoring

• print one startup block: job id, model, input, output, limits
• monitor loop every 120s by default
• do not print per-step noisy diagnostics unless requested
• write #SBATCH -o/-e and code-generated run logs to project-local directories by default

1. Saving and Recovery

• pass save controls to Python entry:
• --save-every
• --save-every-batches
• --save-every-seconds
• --progress-json
• Python entry must save atomically and flush on signals
• For training jobs, checkpoint logic must satisfy exact-training-resume-guard; model-only saves do not count as resumable training checkpoints.

Python Entry Requirements

• Add autosave policy driven by rows/batches/time.
• Save output CSV atomically.
• Save <output>.progress.json with current language/column/counters.
• Install signal handlers for graceful early termination with forced save.
• Process in batches and write translated content incrementally to memory + periodic disk flush.

Default Parameters

• MONITOR_INTERVAL=120
• SAVE_EVERY=200
• SAVE_EVERY_BATCHES=10
• SAVE_EVERY_SECONDS=120
• LOG_EVERY_BATCHES=10

Definition of Done

A job script is only done when all are true:

• preflight checks implemented
• scratch-only heavy paths enforced
• monitor loop exists and is cleanup-safe
• Python entry supports autosave and signal-safe final save
• training jobs support exact resume with full optimizer/scheduler/RNG/framework state
• output CSV and progress JSON are written during run (not only at end)
• code-generated logs and lightweight experiment artifacts stay inside the repository unless the user explicitly requested another path
• checkpoint trees and checkpoint-writing output directories stay under scratch
• large artifacts and large result directories stay under scratch
• model/tokenizer/processor repos are pre-cached under scratch and the run executes offline by default