kwongfuk/codex-skills/palmetto-cpu-request-advisor

Palmetto CPU Request Advisor

Use this skill before submitting or retuning Palmetto jobs when CPU sizing is unclear.

Goal:

• estimate how many CPU cores the workload can really use
• avoid serial jobs requesting many idle cores
• choose a conservative starting CPU count for GPU jobs
• re-check completed runs and shrink or raise CPU requests based on evidence

When To Use

Trigger this skill when the user asks to:

• choose #SBATCH --cpus-per-task or #SBATCH --ntasks
• explain low CPU-Util from jobstats
• size CPUs for GPU training, inference, or dataloader-heavy jobs
• size CPUs for serial Python, MATLAB, R, shell, or single-process jobs
• decide whether a current CPU request is wasteful before sbatch

Workflow

1. Identify the workload shape:

• serial single-process: one Python process, one script, no worker pool, no MPI
• threaded: OpenMP, BLAS, PyTorch intra-op threads, JVM threads, or similar
• multi-process: dataloader workers, multiprocessing pools, MPI ranks, task farms

1. Check prior evidence first when available:

jobstats <jobid>

• If CPU-Util is close to 100% / allocated CPU-cores, treat the job as effectively single-core.
• Aim for about 90%+ of the allocated CPU capacity for sustained CPU-bound jobs.
• If the job used far less CPU than requested, lower the next submission.

1. If no prior run exists, choose a conservative starting point:

• CPU-only serial jobs: --nodes=1, --ntasks=1, --cpus-per-task=1
• CPU-only threaded jobs: start from the real thread count, not a guess
• 1-GPU jobs: start around 4-8 CPU / GPU
• 2+ GPU single-node jobs: start around 4-8 CPU / GPU, then cap to what the input pipeline can actually feed
• Dataloader-heavy jobs: count the actual worker processes and a small margin for the main process

1. Distinguish ntasks from cpus-per-task correctly:

• Use higher ntasks when the code launches multiple independent processes or MPI ranks.
• Use higher cpus-per-task when one task uses threads.
• Do not increase both unless the workload truly uses both processes and threads.

1. After the first run, re-check and retune:

• If GPUs are underutilized and CPU usage is high, increase CPUs modestly.
• If CPU usage is low, lower CPUs before the next run.
• If the job is serial, keep it at 1 CPU unless profiling proves otherwise.

Heuristics

• Treat serial jobs as 1 CPU by default.
• Treat extra CPUs as justified only by observed threads, worker processes, or measured throughput gains.
• For PyTorch jobs, remember that num_workers, tokenization, decoding, and data staging can need extra CPUs, but many training loops still do not need large CPU counts.
• Prefer smaller CPU requests first on busy partitions; they usually start sooner and waste less.
• If the user already knows the code is single-core, say so plainly and recommend --cpus-per-task=1.
• If the user also needs to know whether the reduced request can start now, switch to $palmetto-slurm-fit-check.

Notes

• CPU-Util well below the allocated capacity is a resizing signal, not a success metric.
• One completed pilot run with jobstats is usually more reliable than guessing from the script alone.
• Keep the recommendation explicit: say what CPU count to request next and why.