kwongfuk/codex-skills/palmetto-slurm-fit-check

Palmetto Slurm Fit Check

Use this skill before submitting or retuning a Palmetto GPU job.

Goal:

• inspect the current node fragmentation
• decide whether the requested CPU/GPU/memory tuple can start now
• recommend a lower CPU request when GPUs are available but CPUs are fragmented
• tell the user whether to submit now, lower CPUs, or wait

When To Use

Trigger this skill when the user asks to:

• submit or resubmit a Palmetto GPU job
• tune #SBATCH --cpus-per-task for H100/A100/H200 jobs
• check whether a job can run now
• find a better CPU/GPU ratio before sbatch
• explain why a GPU job is pending even though some GPUs look free

Workflow

1. Read the target request from the sbatch script when possible.
2. Run the fit checker before any submission:

python /home/gguo/.codex/skills/palmetto-slurm-fit-check/scripts/check_slurm_fit.py \
  --sbatch-script /path/to/job.sbatch

1. Interpret the verdict:

• CAN_RUN_NOW: the current request fits at least one node now
• LOWER_CPU_AND_RESUBMIT: GPUs and memory fit, but CPUs are too fragmented; lower --cpus-per-task
• WAIT_OR_REDUCE_MEMORY: GPUs fit, memory does not
• WAIT_FOR_GPUS: not enough free GPUs of the requested type

1. Do not submit immediately when the verdict is LOWER_CPU_AND_RESUBMIT.

• Patch the script first.
• Prefer the script's recommended CPU count over the original request.

1. After patching, validate with:

sbatch --test-only /path/to/job.sbatch

1. Only then submit or resubmit the job.

Heuristics

• Treat CPU fragmentation as a first-class blocker, not just GPU count.
• Default conservative CPU targets:
• h100, h200, a100, v100: about 12 CPU / GPU
• p100, k40, k20, gtx_1080: about 8 CPU / GPU
• Round CPU recommendations down in steps of 4 unless the user asks otherwise.
• If the script can fit with fewer CPUs on a partially free node, tell the user that explicitly instead of keeping an inflated CPU request.

Notes

• The checker handles the local scontrol shared-library issue automatically by adding a temporary libhistory.so.7 shim when needed.
• The checker is intended for single-node GPU jobs. If a script requests multiple nodes, say that the fit result is only approximate and inspect manually.
• Keep the decision user-visible: explain whether the job can run now and why.