kwongfuk/codex-skills/palmetto-conda-scratch-guard

Palmetto Conda Scratch Guard

Rules

• Treat /home/$USER as the source tree area: code, configs, small manifests, and standard lightweight experiment results may live there.
• Treat /scratch/$USER as the heavy-storage area: envs, caches, datasets, model weights, container images, checkpoints, raw intermediate outputs, and any large file or large directory go there.
• Create new environments under /scratch/$USER/envs/<env_name>.
• Store Hugging Face model cache under /scratch/$USER/.hf_cache.
• Store dataset downloads under /scratch/$USER/data.
• Store model weights and large generated artifacts under /scratch/$USER/models or another scratch-backed project path.
• Store container images under /scratch/$USER/containers.
• Store pip cache under /scratch/$USER/.pip_cache.
• In Codex/agent-run Slurm jobs, do not assume $USER matches the real Palmetto account. If the session user can be coder, use an explicit absolute scratch root such as /scratch/<real_user> or a validated SCRATCH_ROOT variable instead of deriving critical paths from $USER.
• Do not create heavy Conda environments under /home/$USER unless the user explicitly requests it.
• Do not place large caches, downloaded models, dataset shards, checkpoints, or other large artifacts under /home/$USER.
• If an env path is missing, create /scratch/$USER/envs first.
• Keep login node usage light; prefer compute allocation for heavy install/build workloads.
• Set PYTHONNOUSERSITE=1 for env checks to avoid accidental package shadowing from ~/.local.

Workflow

1. Run preflight checks:

bash scripts/palmetto_preflight.sh

1. Create environment in scratch:

mkdir -p /scratch/$USER/envs
conda create -p /scratch/$USER/envs/<env_name> python=3.10 -y

1. Activate by absolute path:

conda activate /scratch/$USER/envs/<env_name>

1. Install packages.

1. Set cache paths to scratch before model/data/container downloads:

mkdir -p /scratch/$USER/{.hf_cache,.pip_cache,data,containers,models}
export HF_HOME=/scratch/$USER/.hf_cache
export TRANSFORMERS_CACHE=$HF_HOME/transformers
export HUGGINGFACE_HUB_CACHE=$HF_HOME/hub
export PIP_CACHE_DIR=/scratch/$USER/.pip_cache
export PYTHONNOUSERSITE=1

Decision Guide

• If preflight reports NODE_CLASS=login and task is heavy (large pip/conda builds, CUDA extensions, model runtime setup), move to an allocated compute node first (salloc or srun --pty bash).
• If task includes pulling large models/datasets/containers, run on compute allocation and keep all targets under /scratch/$USER.
• If an experiment produces only standard summary artifacts such as final tables, plots, or compact metrics, they may stay with the code under /home/$USER.
• If an output is large, resumable, or expensive to regenerate, keep it under /scratch/$USER even if the final summarized result is later copied back to the project tree.
• If nvidia-smi is missing or no GPUs are visible, check node class first. Login nodes often have no GPUs.
• If torch.cuda.is_available() is false but GPUs exist, verify CUDA toolkit/driver compatibility and env package build.

References

• See references/checklist.md for a compact troubleshooting checklist.