zhangyanbo/hpc-skills/epfl-haas

EPFL RunAI ("Haas") GPU Cluster Operations

Deploy, submit, monitor, and retrieve results on an EPFL RunAI/Kubernetes GPU cluster over SSH, on the user's behalf. Top priority: never do anything that could violate cluster policy or draw administrator attention (see "Compliance red lines"). When in doubt, take the slower, by-the-book path — the user's account is not worth any shortcut.

This is a RunAI/Kubernetes environment, not SLURM: there is no sbatch/squeue/sinfo. Jobs are Kubernetes pods submitted and inspected via the runai CLI, scheduled across GPU node pools rather than SLURM partitions.

0. Preflight — run this before any Haas work

All connection details live on the local machine in ~/.config/epfl-haas/config (shell syntax, KEY=value). This skill contains no account information; the config file is the single source of truth. Start every Haas session with:

bash <this-skill-dir>/scripts/preflight.sh

Branch on its output:

• READY host=... node=... — SSH to the control-plane host works; proceed.

source the config file to get:

• HAAS_HOST — ssh target (an alias or user@host) for the control-plane

host used to run runai commands. Use it everywhere: ssh "$HAAS_HOST" ...

• HAAS_USER — the user's login
• HAAS_PROJECT — the RunAI project every submission is scoped to; ensure

it is active (runai config project "$HAAS_PROJECT") or pass -p "$HAAS_PROJECT" per command

• HAAS_PVC_ROOT — durable PVC-backed project storage root (a project's

own remote path, if recorded in that project's CLAUDE.md, takes precedence)

• Note: a READY preflight only confirms SSH — it does not confirm

runai auth. RunAI's own login/OAuth token can expire independently; see §2.

• NO_CONFIG — first use on this machine; run "Guided setup" below.
• NO_PASSWORDLESS — key-based login is broken, or the network route (VPN)

to the control-plane host is unreachable. Stop all automated Haas operations. Tell the user: passwordless SSH is a prerequisite for automation — set up a key, confirm VPN/network routing, then retry. Never type passwords for the user or try to work around authentication.

Guided setup (only on NO_CONFIG)

1. Ask the user for their control-plane SSH alias/host and RunAI project

name — never guess account information.

1. Test passwordless login:

ssh -o BatchMode=yes -o ConnectTimeout=10 <host> 'echo OK && hostname'.

1. On success, write ~/.config/epfl-haas/config (template in the header of

scripts/preflight.sh, including HAAS_PROJECT) and chmod 600 it. Verify the runai CLI is actually available and authenticated where the skill will run it: ssh <host> 'command -v runai && runai whoami' — if runai is missing on the SSH host, the user runs the CLI locally and this must be sorted out with them before any automation.

1. On failure, follow NO_PASSWORDLESS above; do not write

HAAS_PASSWORDLESS=yes.

1. Compliance red lines (each one is a hard constraint)

1. The control-plane host is for light operations only: ls / cat /

runai list / editing small files / short scp. Anything that burns CPU, memory, or heavy IO — dependency installs, extracting large archives, bulk deletes, data processing, running any program — must go through a RunAI pod, not the control-plane shell.

1. Every Haas action must trace back to an explicit user request. Once

the user says "run X on Haas", the deploy → submit → monitor → retrieve chain for that task can run autonomously; actions outside that scope (touching other projects' namespaces, deleting unrelated jobs) are off-limits.

1. Destructive operations require a confirmed list first: bulk rm on

PVC storage, overwriting existing checkpoints/outputs, deleting jobs not submitted in this task, releasing a shared/opportunistic allocation another task may still need.

1. Rate-limit polling: status-check loops at ≥ 60-second intervals; batch

runai describe/runai list calls in one ssh round-trip where possible instead of many separate calls.

1. Stay well below storage ceilings. PVC usage can grow quietly, e.g.

from per-run virtualenv/cache proliferation — before large writes or many new environments, sanity-check actual physical usage (du, stat for hard-link sharing) rather than assuming each new env is fully additional.

1. No restricted data on the cluster. Never store or enter passwords/API

keys on the user's behalf — those belong in a project .env or mounted secret, not inline in runai submit -e ... flags when avoidable. Treat runai describe job output as potentially sensitive: it can echo the original submit command and environment variables — don't paste it raw into chats, docs, or commits.

2. RunAI auth is separate from SSH

A working READY preflight only proves SSH to the control-plane host works. runai itself typically needs its own login flow (often OAuth-based), and that token can expire independently of the SSH session. Before trusting a runai command's output:

ssh "$HAAS_HOST" 'runai whoami'

If this fails with an auth/token error (e.g. invalid_grant or similar), the fix is a runai login (or the cluster's current equivalent) — report the exact re-login step to the user rather than continuing with what would be misleading job-status analysis on a stale/failed auth session.

3. Status checks (when the user asks "how are my jobs doing?")

ssh "$HAAS_HOST" 'runai list jobs; echo ---; runai whoami'

Failure triage order: runai describe job <name> events (distinguish node-pool saturation from project fair-share/quota, or a plain code error) → runai logs <name> tail → re-check runai whoami if the describe/logs output looks stale or the job never left Pending. Treat ContainerCreating, long image pulls, and image-cache misses as node/image startup problems, not code failures.

4. Task routing

• **Deploying code / transferring files / setting up environments (conda, uv)

/ PVC storage layout** → read references/deploy.md first.

• **Submitting RunAI jobs / choosing node pools / interactive vs batch pods /

debugging jobs** → read references/runai.md first.

• Both at once (the common "run X on Haas") → read both, execute in

deploy → runai order.

• **Something in this skill turned out wrong or missing, or the user says

"file that as an issue"** → §8 (draft an issue for the skill's repository; file only after the user confirms).

5. Using Haas inside a loop (submit → wait → retrieve → iterate)

Standard loop skeleton:

1. Deploy changed files (full dependency-chain check, see deploy.md), then

submit (see runai.md).

1. Wait in the background, polling at ≥ 60s intervals:

``bash ssh "$HAAS_HOST" 'runai list jobs | grep -c Running' ` Then immediately classify with runai list jobs / runai describe job` — Succeeded vs Failed vs still-Pending. An empty "Running" count does not by itself mean success.

1. Retrieve results (create the local destination first — rsync does not create

a missing parent directory, and then exits 0 having copied nothing): ``bash mkdir -p ./results && rsync -azP "$HAAS_HOST":<PVC-path>/results/ ./results/ && ls ./results | wc -l `` Incremental and safe to re-run; the trailing count is the check that files actually arrived — never trust a clean exit alone. Assumes SSH access to a path that mounts the same PVC, e.g. the control-plane host or an interactive pod's exposed path.

1. Analyze locally → adjust code/parameters → back to 1. Re-run only the

missing tasks using the idempotent submit pattern (runai.md, "Idempotent job-matrix submission") — that pattern is what makes the whole loop safely re-entrant.

6. Key cluster facts (quick recall; details in references/)

• Scheduler: RunAI on Kubernetes, reached via the runai CLI from a

control-plane host — no sbatch/squeue/sinfo equivalents.

• Persistent storage is PVC-backed; the pod/container root filesystem does

not survive pod restarts — anything that must persist (code, venvs, data, checkpoints) belongs on the PVC-backed path.

• Node pools stand in for SLURM partitions, but "default" does not

necessarily mean "any available GPU" — it can effectively pin to one GPU class. Node-pool names, GPU classes, and quota (deserved vs opportunistic) are cluster/project-specific; verify live rather than assuming a fixed mapping from another project.

• runai auth (login/OAuth token) is separate from SSH auth to the

control-plane host and can expire independently — see §2.

7. Off-campus / VPN access (knowledge, not an operation)

This section is background to explain when the user asks about connection problems — it is not something this skill configures or performs on its own. The skill always just uses $HAAS_HOST; whether that route needs a VPN is the user's local SSH client configuration, entirely outside the skill's operational scope. Check current institutional IT documentation for the up-to-date VPN/SSH requirements, since these evolve independently of this skill.

8. Reporting problems with this skill

This skill lives in the public repository <https://github.com/Zhangyanbo/hpc-skills> (directory skills/epfl-haas/) and is maintained from real usage: when something in it turns out to be wrong or missing, that is worth an issue at <https://github.com/Zhangyanbo/hpc-skills/issues>. Typical triggers, noticed while doing a task:

• a documented command failed or behaved differently from what the skill says

(wrong module name, flag, path, limit);

• a cluster fact here is stale (partition limits, quotas, hostnames, tool

versions);

• a gap that caused an avoidable detour — something you had to discover the

hard way that the skill should have said up front.

Procedure:

1. Finish the user's task first; collect evidence as you go (the exact

command, its actual output / exit code, the skill file and section that was wrong or silent).

1. Draft the issue in English: title epfl-haas: <one-line symptom>; body with

which file/section, what the skill says, what actually happened, cluster-side evidence, and a suggested fix. Concise and concrete.

1. De-sensitize the text — the repository is public. No usernames /

username, ssh aliases, jump-host details, personal or lab paths: write <username>, $HAAS_HOST, <PVC-root>/... instead. Job IDs and version numbers are fine.

1. Show the draft to the user and file it only with their confirmation

(opening an issue is a public action): ``bash gh issue create --repo Zhangyanbo/hpc-skills --title "<title>" --body-file <draft.md> ` If gh` is unavailable or the user prefers, hand them the draft to post themselves.

Fixes are also welcome as pull requests (see the repository README).