amd-agi/primus/slurm-training-node-validation

SLURM Training Node Validation

Launch short training jobs on groups of idle nodes to determine which nodes are healthy enough to run real workloads.

Prerequisites

Use the slurm-idle-node-check skill first (or manually) to get the list of idle, healthy nodes. If the user has not provided a nodelist, obtain one using the slurm-idle-node-check skill's Step 1 (filter for exactly idle state).

Workflow

Step 0: Pre-flight Checks

Before launching any training, resolve these runtime parameters:

HF_TOKEN

Check if HF_TOKEN is set in the environment:

echo "${HF_TOKEN:-NOT_SET}"

If it is NOT_SET or empty, ask the user to provide it using AskQuestion or a conversational prompt. Do NOT proceed without a valid HF_TOKEN.

Partition

Determine the SLURM partition to use:

1. Check if amd-aig exists:

``bash sinfo -h -o "%P" | tr -d '*' | sort -u ``

1. If amd-aig is in the list, use it.
2. If amd-aig is NOT in the list:

• If there is exactly one partition available, use it automatically.
• If there are multiple partitions, use AskQuestion to let the user pick one.

Store the resolved partition in $PARTITION for use in the launch command.

Runner Script Path

The runner script path is relative to the workspace root, not hardcoded:

RUNNER_SCRIPT="$(pwd)/runner/primus-cli-slurm.sh"

Verify it exists before proceeding:

if [ ! -f "$RUNNER_SCRIPT" ]; then
  echo "ERROR: runner/primus-cli-slurm.sh not found at $RUNNER_SCRIPT"
  exit 1
fi

Step 1: Get Idle Nodes and Form Groups

1. If the user provides a nodelist, use it. Otherwise, run:

``bash sinfo -h -o "%P %T %N" | awk '$2 == "idle" {print $3}' ` Then expand with scontrol show hostnames <nodelist>` to get individual hostnames.

1. If multiple partitions have idle nodes, use AskQuestion to let the user pick a partition.

1. Split nodes into groups of 4. If the total is not a multiple of 4, the remaining nodes (fewer than 4) form an incomplete group — skip that group and inform the user which nodes were excluded.

1. Display the grouping plan to the user before launching. Example:

`` Group 1: node01,node02,node03,node04 Group 2: node05,node06,node07,node08 Group 3: node09,node10,node11,node12 Skipped (incomplete group): node13 ``

Step 2: Launch Training Jobs

For each group, launch a training job in the background with a dedicated log file.

Directory Structure

Use ./output/ relative to the workspace root. The timestamp is generated once at the start and reused for all groups:

TIMESTAMP=$(date +%Y%m%d_%H%M%S)
BASEDIR="./output/slurm-training-node-validation-${TIMESTAMP}"

For each group $i, first compress the 4 hostnames into SLURM bracket notation, then create the group directory and marker file:

COMPRESSED_NODELIST=$(scontrol show hostlist "node1,node2,node3,node4")
# e.g. "node-[1628,1691,1697,1707]"
 
GROUPDIR="${BASEDIR}/group_${i}"
mkdir -p "$GROUPDIR"
 
# Create nodelist marker file — filename and content both use the SLURM-compressed nodelist
echo "${COMPRESSED_NODELIST}" > "${GROUPDIR}/group_${i}_nodelist_${COMPRESSED_NODELIST}"

The `$COMPRESSED_NODELIST` must always be in SLURM prefix-compressed format (e.g. node-[1628,1691,1697,1707]), NOT individual hostnames joined by commas. Always use scontrol show hostlist to produce it.

The resulting directory layout looks like:

./output/slurm-training-node-validation-20260316_201333/
├── group_1/
│   ├── log_group_1.txt                                          # training log
│   ├── group_1_nodelist_node-[1628,1691,1697,1707]  # content: the compressed nodelist
│   └── success  OR  failed                                      # result file (created after job finishes)
├── group_2/
│   ├── log_group_2.txt
│   ├── group_2_nodelist_node-[1711,1713,1719,1723]
│   └── success  OR  failed
└── group_3/
    ├── ...

Launch Command

For each group (index $i, nodelist $NODELIST), run:

(
  export SLURM_NODELIST='<nodelist_for_this_group>'
  export USING_AINIC=1
  export NCCL_IB_HCA="ionic_0:1,ionic_2:1,ionic_3:1,ionic_4:1,ionic_5:1,ionic_7:1,ionic_8:1,ionic_9:1"
  export GLOO_SOCKET_IFNAME=ens9np0
  export NCCL_SOCKET_IFNAME=ens9np0
  export CLEAN_DOCKER_CONTAINER=1
  export GPU_MAX_HW_QUEUES=4
  export HF_TOKEN="${HF_TOKEN}"
 
  bash "${RUNNER_SCRIPT}" \
    -N 4 --time=48:00:00 --partition="${PARTITION}" \
    -- --image docker.io/tasimage/primus:pr-563-ainic --clean \
    -- train pretrain \
    --config examples/megatron/configs/MI355X/deepseek_v2_lite-BF16-pretrain.yaml \
    --num_layers 10 --moe_layer_freq 1 --train_iters 5 \
    --micro_batch_size 2 --global_batch_size 64 \
    --cross_entropy_fusion_impl te --cross_entropy_loss_fusion True \
    --recompute_num_layers 0 --recompute_granularity full \
    --recompute_method block --disable_last_saving True \
    --profile False --use_pytorch_profiler False \
    --profile_step_end 7 --profile_step_start 6
) > "${GROUPDIR}/log_group_${i}.txt" 2>&1 &

Important details:

• Replace <nodelist_for_this_group> with the compressed nodelist for this group. Use scontrol show hostlist node01,node02,node03,node04 to compress it, or just use the comma-separated list.
• ${HF_TOKEN} — resolved in Step 0 (from environment or user input).
• ${RUNNER_SCRIPT} — resolved in Step 0 as $(pwd)/runner/primus-cli-slurm.sh.
• ${PARTITION} — resolved in Step 0 (auto-detected or user-selected).
• Each group runs in a subshell so environment variables don't leak between groups.
• Capture the PID of each background job: PIDS[$i]=$!
• Log file: log_group_1.txt, log_group_2.txt, etc. (1-indexed)
• The nodelist marker file name embeds the compressed nodelist (e.g. group_1_nodelist_node-[1628,1691,1697,1707]). The file content is the compressed nodelist string itself.

After launching all groups, print:

Launched <N> training groups. Output directory: $BASEDIR
Waiting for all jobs to complete...

Step 3: Wait and Monitor

1. Use wait to wait for all background PIDs to finish:

``bash for i in "${!PIDS[@]}"; do wait ${PIDS[$i]} 2>/dev/null EXIT_CODES[$i]=$? done ``

1. Alternatively, poll logs periodically (every 60 seconds) to provide progress updates to the user. Use the Shell tool with appropriate block_until_ms to monitor.

1. Early failure detection: While polling, if a log contains Exited with exit code 1 but does NOT contain iteration 3, the job has already failed — no need to wait further. Mark it as failed immediately.

1. Timeout: If a job hasn't finished after 5 minutes:

• Kill the background process: kill ${PIDS[$i]} 2>/dev/null
• Find and cancel any SLURM jobs launched by this group. Search for jobs on the group's nodes:

``bash # Find SLURM job IDs associated with this group's nodes squeue -u "$USER" -h -o "%i %N" | while read jobid nodes; do # Check if any of this group's nodes appear in the job's nodelist if echo "$nodes" | grep -qF "<any_node_from_this_group>"; then scancel "$jobid" fi done ` Alternatively, if the training script outputs a SLURM job ID in the log, extract and cancel it: `bash JOB_ID=$(grep -oP 'Submitted batch job \K\d+' "${GROUPDIR}/log_group_${i}.txt" 2>/dev/null || true) if [ -n "$JOB_ID" ]; then scancel "$JOB_ID" fi ``

• Mark the group as failed with reason "Timeout after 5 minutes".

Step 4: Check Results and Write Status Files

For each group, determine the result by checking the log file:

HAS_ITER3=$(grep -c "iteration        3" "${GROUPDIR}/log_group_${i}.txt" 2>/dev/null || echo 0)
HAS_EXIT1=$(grep -c "Exited with exit code 1" "${GROUPDIR}/log_group_${i}.txt" 2>/dev/null || echo 0)

Decision logic:

1. If HAS_ITER3 > 0 → PASS (regardless of exit code).
2. If HAS_ITER3 == 0 and HAS_EXIT1 > 0 → FAIL (process exited with error before reaching iteration 3).
3. If HAS_ITER3 == 0 and HAS_EXIT1 == 0 and job timed out → FAIL (timeout).

Note: The pattern iteration 3 uses 8 spaces between "iteration" and "3". This is the exact pattern printed by Megatron-LM during training. Use grep with the exact string.

On success (pattern found):

Create an empty success file in the group directory:

touch "${GROUPDIR}/success"

On failure (pattern not found):

Extract error context and write a failed file with error details:

# Collect last 50 lines for error context
ERROR_CONTEXT=$(tail -50 "${GROUPDIR}/log_group_${i}.txt")
 
# Look for known error patterns
ERRORS=""
for pattern in "RuntimeError" "NCCL error" "OutOfMemoryError" "OOM" "Timeout" "ConnectionError" "CalledProcessError" "CUDA error" "Exited with exit code" "Error encountered progressing operation=Connect" "couldn't chdir to"; do
  match=$(grep -m1 "$pattern" "${GROUPDIR}/log_group_${i}.txt" 2>/dev/null || true)
  if [ -n "$match" ]; then
    ERRORS="${ERRORS}${match}\n"
  fi
done
 
# Write failed file with error info and possible cause
cat > "${GROUPDIR}/failed" <<FAILEOF
=== Training Validation Failed ===
Group: ${i}
Nodelist: ${COMPRESSED_NODELIST}
Log: ${GROUPDIR}/log_group_${i}.txt
 
--- Detected Errors ---
${ERRORS:-No specific error pattern detected.}
 
--- Possible Cause ---
$(if echo "$ERRORS" | grep -q "NCCL"; then
    echo "NCCL communication failure — possible network issue or faulty NIC on one of the nodes."
  elif echo "$ERRORS" | grep -q "OOM\|OutOfMemory"; then
    echo "GPU out of memory — possible hardware issue or memory leak on one of the nodes."
  elif echo "$ERRORS" | grep -q "Timeout"; then
    echo "Operation timed out — possible node hang or network partition."
  elif echo "$ERRORS" | grep -q "CUDA error"; then
    echo "CUDA error — possible GPU hardware fault on one of the nodes."
  elif echo "$ERRORS" | grep -q "couldn't chdir to"; then
    echo "Working directory not accessible — network filesystem likely not mounted on one or more nodes."
  else
    echo "Training did not reach iteration 3. Check the log file for details."
  fi)
 
--- Last 50 Lines of Log ---
${ERROR_CONTEXT}
FAILEOF

If the job was killed due to timeout, write:

cat > "${GROUPDIR}/failed" <<FAILEOF
=== Training Validation Failed ===
Group: ${i}
Nodelist: ${COMPRESSED_NODELIST}
 
--- Detected Errors ---
Timeout after 5 minutes.
 
--- Possible Cause ---
Job exceeded the 5-minute timeout. Nodes may be hung or experiencing severe performance issues.
FAILEOF

Important: If a group has a non-zero exit code but the success pattern IS found in the log, still create success (not failed), because training may exit non-zero after completing.

Step 5: Display Results

Always write the summary in English, regardless of the conversation language.

Results Table

## Training Validation Results
 
| Group | Nodelist | Status | Error |
|-------|----------|--------|-------|
| 1 | node[01-04] | PASS | - |
| 2 | node[05-08] | FAIL | NCCL timeout on node06 |
| 3 | node[09-12] | PASS | - |

Summary

## Summary
 
-Total groups tested: <N>
-Passed: <P>
-Failed: <F>
-Output directory: <BASEDIR>
 
### Passed Nodelist (training-ready)
<SLURM-format compressed nodelist, e.g. node-[1628,1691,1697,1707,1711,1713,1719,1723]>
 
### Failed Nodelist
<SLURM-format compressed nodelist, e.g. node-[1730,1735,1741,1749]>

Generating SLURM-format Compressed Nodelists

Collect all individual hostnames for passed (or failed) groups, then compress into SLURM nodelist format:

scontrol show hostlist <comma-separated-hostnames>

Example: if passed nodes are node-1628,node-1691,node-1697,node-1707,node-1711,node-1713, then: