kwongfuk/codex-skills/palmetto-apptainer-libcuda-fix

Palmetto Apptainer libcuda Fix

Use this skill when a Palmetto GPU job fails inside Apptainer with messages like:

• libcuda.so cannot found
• Possible files are located at ['/lib64/libcuda.so.1']
• vLLM engine core startup fails before the model becomes ready

Goal

Repair container-side CUDA driver discovery without hand-waving. Do not just append more paths to LD_LIBRARY_PATH and hope. Make the missing soname visible at the exact path the containerized workers need, then prove it with a short smoke run before resubmitting the real job.

Required workflow

1. Confirm the failure signature.

• Read the Slurm stdout/stderr or service log.
• Verify the root cause is specifically missing libcuda.so inside the containerized worker path.
• If the failure is a different CUDA error, do not use this fix blindly.

1. Patch the sbatch launcher.

• Add a scratch-local compatibility directory, for example:
• CUDA_COMPAT_DIR="$JOB_TMP_ROOT/cuda_compat"
• CUDA_COMPAT_LIB="$CUDA_COMPAT_DIR/libcuda.so"
• Create the symlink on the host:
• mkdir -p "$CUDA_COMPAT_DIR"
• ln -sfn /lib64/libcuda.so.1 "$CUDA_COMPAT_LIB"
• Bind that exact file into the container at the missing soname path:
• CUDA_COMPAT_BIND="$CUDA_COMPAT_LIB:/lib64/libcuda.so"
• APPTAINER_GPU_BINDPATH="${APPTAINER_BINDPATH},${CUDA_COMPAT_BIND}"
• For the GPU-facing apptainer exec --nv --cleanenv calls, use:
• --bind "$APPTAINER_GPU_BINDPATH"
• --env LD_LIBRARY_PATH="/lib64:$SCRATCH_ROOT/envs/vllm_env/lib"
• Keep non-GPU preflight calls on the normal bind path unless they also need the driver.

1. Add a real preflight before the main server starts.

• Run a short apptainer exec --nv --cleanenv Python check that does all of:
• ctypes.CDLL("libcuda.so")
• ctypes.CDLL("libcuda.so.1")
• optional torch.cuda.is_available()
• Fail fast here instead of waiting for a long vLLM startup timeout.

1. Run a smoke validation job.

• Submit a short one-GPU Slurm job that uses the same bind strategy as the real launcher.
• The smoke must confirm both parent and child Python processes can load:
• libcuda.so
• libcuda.so.1
• On Palmetto, prefer a short V100 or P100 smoke if H100 priority is slow. This checks driver visibility, which is the point of the fix.

1. Only then resubmit the real job.

• Reuse the patched launcher.
• Preserve any previous environment overrides such as MODES, SKIP_EXISTING_SCENARIOS, or custom output roots.
• Watch the new run until it clears the previous failure stage. Do not claim success just because the job is queued.

Patch pattern

Use this pattern in the sbatch file:

CUDA_COMPAT_DIR="$JOB_TMP_ROOT/cuda_compat"
CUDA_COMPAT_LIB="$CUDA_COMPAT_DIR/libcuda.so"
CUDA_COMPAT_BIND="$CUDA_COMPAT_LIB:/lib64/libcuda.so"
APPTAINER_GPU_BINDPATH="${APPTAINER_BINDPATH},${CUDA_COMPAT_BIND}"
 
mkdir -p "$CUDA_COMPAT_DIR"
ln -sfn /lib64/libcuda.so.1 "$CUDA_COMPAT_LIB"
 
apptainer exec --nv --cleanenv \
  --bind "$APPTAINER_GPU_BINDPATH" \
  --env LD_LIBRARY_PATH="/lib64:$SCRATCH_ROOT/envs/vllm_env/lib" \
  "$APPTAINER_IMAGE" \
  "$VLLM_PYTHON" - <<'PY'
import ctypes
ctypes.CDLL("libcuda.so")
ctypes.CDLL("libcuda.so.1")
print("container_cuda_driver_ok")
PY

Use the same APPTAINER_GPU_BINDPATH and LD_LIBRARY_PATH in the real server launch.

Acceptance criteria

Do not consider the fix complete until all are true:

• the smoke job exits COMPLETED
• smoke stdout shows both parent and child process driver loads succeeded
• the real benchmark job has been resubmitted
• the real job advances past the old libcuda.so cannot found stage

Notes

• This is a container driver-visibility fix, not a general CUDA fix.
• The key move is binding a host-side symlink into /lib64/libcuda.so inside the container, because some worker paths look for that soname directly.
• If the workload later fails with OOM, model config, or networking errors, that is a new problem and should be debugged separately.