amdresearch/aup-learning-cloud/troubleshoot-aup-learning-cloud

Troubleshoot AUP Learning Cloud

Find the root cause of a failing deployment from runtime evidence, name it, and point at the fix — without thrashing. Gather state first, match the symptom to a known cause, change one thing, re-check. The full symptom → cause → checks matrices live in [reference.md](reference.md).

Prerequisites

• Access to the cluster (kubectl, the right KUBECONFIG) and/or the service

machine (for PXE/host issues).

• A checkout of aup-learning-cloud for config cross-checks.
• The deploy skill's $DEPLOY_SCRIPTS/detect_cluster.sh is a fast way to snapshot

nodes, GPU labels, storage classes, and the device plugin/labeller state.

From any checkout directory, define REPO_ROOT="$(git rev-parse --show-toplevel)" and DEPLOY_SCRIPTS="$REPO_ROOT/skills/deploy-aup-learning-cloud/scripts". For an installed plugin, define DEPLOY_SKILL_DIR as the absolute directory containing the loaded deploy skill's SKILL.md, then set DEPLOY_SCRIPTS="$DEPLOY_SKILL_DIR/scripts".

Method (don't thrash)

1. Scope it. Which layer is failing — netboot, node join, GPU scheduling,

storage, or auth? One layer at a time.

1. Gather evidence before acting.

``bash kubectl get nodes -o wide kubectl get pods -A | grep -Ev 'Running|Completed' kubectl describe pod -n jupyterhub <pod> # Events explain Pending/ImagePull "$DEPLOY_SCRIPTS/detect_cluster.sh" # from the deploy skill ``

1. Match to a cause using the reference.md matrices.
2. Change one thing, then re-check the same evidence. Do not stack

speculative changes. After ~4 failed attempts with no new evidence, stop and report what you observed and the most likely next step.

1. Hand off the fix to the right skill (below) rather than improvising.

Where each fix lives

First checks by layer

• Netboot: systemctl status dnsmasq nfs-kernel-server apache2,

journalctl -u dnsmasq, firmware boot order + Secure Boot, TFTP files in /srv/tftp.

• Node join: systemctl status k3s-agent, journalctl -u k3s-agent,

hostname/api_endpoint/token, curl http://<SERVICE_IP>:8080/k3s/token.

• GPU: kubectl get ds -A | grep amd,

kubectl describe node <n> | grep amd.com/gpu, then compare to custom.accelerators.*.nodeSelector.

• Storage: kubectl get pvc -A, provisioner logs, showmount -e <NFS>,

/etc/exports.

• Auth: Hub logs (kubectl logs -n jupyterhub deploy/hub), custom.auth

provider flags, and the GitHub OAuth callback URL. Check resource visibility separately through custom.teams.mapping and the user's fallback group.

Safety

Evidence-first and read-only by default. Stop and get explicit confirmation before any state change, especially:

• kubectl delete node <name> (clears a stale node object — debugging only).
• helm uninstall, helm rollback, or recreating any PVC (data loss).
• pb-k3s-reset.yml (whole cluster or --limit <node>).
• Rebuilding the PXE rootfs under running agents (pxe_rootfs_force_rebuild).

Never commit changes or write secrets (k3s token, OAuth secrets, SSH keys) into tracked files while debugging.

Reference

Full symptom → cause → first-checks matrices for netboot, node join, GPU, storage, auth, and kubeconfig, plus the reset/escape hatches: reference.md.