zhangyanbo/hpc-skills/tufts-hpc

Tufts HPC Operations

Deploy, submit, monitor, and retrieve results on the Tufts HPC (SLURM) cluster over SSH, on the user's behalf. Top priority: never do anything that could violate cluster policy or draw administrator attention (see "Compliance red lines"). When in doubt, take the slower, by-the-book path — the user's account is not worth any shortcut.

0. Preflight — run this before any HPC work

All connection details live on the local machine in ~/.config/tufts-hpc/config (shell syntax, KEY=value). This skill contains no account information; the config file is the single source of truth. Start every HPC session with:

bash <this-skill-dir>/scripts/preflight.sh

(ssh may print warnings such as post-quantum key-exchange notices, possibly twice when a jump host is in the path — that is normal noise, not an error.)

Branch on its output:

• READY host=... node=... — connection works; proceed. source the config file

to get:

• HPC_HOST — ssh target (an alias or user@host). Use it everywhere:

ssh "$HPC_HOST" ..., rsync ... "$HPC_HOST":...

• HPC_USER — the user's Tufts UTLN
• HPC_DEFAULT_REMOTE_ROOT — default deployment root on the cluster (a

project's own remote path, if recorded in that project's CLAUDE.md, takes precedence). It is meant to stay a literal ~/... or /cluster/... path. If preflight printed a WARN: about it (typically a local-machine path such as /Users/<me>/... or /home/<me>/..., caused by an unquoted ~ in the config expanding on the local side), do not deploy to that value: fix the config (quote it: HPC_DEFAULT_REMOTE_ROOT="~/research") or use an explicit cluster path.

• HPC_RESEARCH_DIR — research-storage path (may be empty = user has no lab

storage yet; warn before placing large data anywhere)

• NO_CONFIG — first use on this machine; run "Guided setup" below.
• NO_PASSWORDLESS — key-based login is broken. **Stop all automated HPC

operations.** Tell the user: passwordless SSH is a prerequisite for automation — set up a key (ssh-keygen + ssh-copy-id <host>; if a jump host is involved, it needs a key too), then retry. Never type passwords for the user or try to work around authentication.

Guided setup (only on NO_CONFIG)

1. Scan for candidates: grep -B1 -A4 -i 'pax.tufts.edu' ~/.ssh/config.

Prefer an alias whose HostName is login-prod.pax.tufts.edu (the new cluster). login.pax.tufts.edu (no -prod) is the old cluster being retired — do not use it.

1. If nothing is found or multiple candidates exist, ask the user which Host to

use and what their UTLN is. Never guess account information.

1. Test passwordless login:

ssh -o BatchMode=yes -o ConnectTimeout=10 <host> 'echo OK && hostname'.

1. On success, write ~/.config/tufts-hpc/config (template in the header of

scripts/preflight.sh) and chmod 600 it. Quote any ~ in path values (HPC_DEFAULT_REMOTE_ROOT="~/research") — the file is sourced locally and an unquoted ~ would silently become the local home. Also probe for research storage: ssh <host> 'ls -d /cluster/tufts/*/$USER 2>/dev/null' and record it in HPC_RESEARCH_DIR if present.

1. On failure, follow NO_PASSWORDLESS above; do not write HPC_PASSWORDLESS=yes.

1. Compliance red lines (each one is a hard constraint)

1. Login nodes are for light operations only: ls / cat / squeue / sbatch /

editing small files / small scp. Anything that burns CPU, memory, or heavy IO — installing packages, building conda/uv environments, extracting large archives, bulk-deleting big directories, compressing, data processing, running any program — must go through a compute node: ``bash # Interactive (for a sequence of manual steps; QOS caps at 4 hours) ssh -t "$HPC_HOST" 'srun -p batch -t 0-1:00:00 -c 4 --mem=8G --pty bash' # One-shot (for automation: wrap a single heavy command in srun) ssh "$HPC_HOST" 'srun -p batch -t 0-0:30:00 -c 2 --mem=4G bash -c "cd ~/proj && tar xzf data.tar.gz"' `` All nodes share the same storage (home and /cluster), so work done on any compute node is visible everywhere — nodes differ only in compute power. "Borrowing a compute node for chores" has zero downside; don't hesitate.

1. Every HPC action must trace back to an explicit user request. Once the user

says "run X on the HPC", the deploy → submit → monitor → retrieve chain for that task can run autonomously; actions outside that scope (touching other directories, cancelling unrelated jobs) are off-limits.

1. Destructive operations require a confirmed list first: bulk rm,

overwriting existing remote results, scancel on jobs not submitted in this task.

1. Rate-limit polling: status-check loops at ≥ 60-second intervals (don't

hammer squeue). Batch several remote commands into one ssh call (ssh host 'cmd1; cmd2; cmd3') — fewer connections, and much lower latency through a jump host.

1. Stay well below quota ceilings. Before a large submission, check current

load with squeue --me | wc -l. Per-user limits: batch+gpu combined ≤ 250 CPUs / 10 GPUs; preempt ≤ 1000 CPUs / 20 GPUs. Throttle arrays with %N (e.g. %200).

1. No restricted data on the cluster (HIPAA, FERPA, etc.). Never store or

enter passwords on the user's behalf.

2. Status checks (when the user asks "how are my jobs doing?")

Grab everything in one ssh call:

ssh "$HPC_HOST" 'squeue --me; echo ---; sacct -X --starttime today -o JobID,JobName%20,State,Elapsed,ExitCode | tail -30'

Failure triage order: tail of the .err file → sacct State/ExitCode (OOM → more memory, TIMEOUT → more time, NODE_FAIL/PREEMPTED → just resubmit) → seff to see whether resources were undersized.

3. Task routing

• **Deploying code / transferring files / setting up environments (conda, uv) /

storage & quota issues** → read references/deploy.md first.

• **Writing SLURM scripts / submitting jobs / array-job experiment matrices /

choosing partitions / debugging jobs** → read references/slurm.md first.

• Both at once (the common "run X on the HPC") → read both, execute in

deploy → slurm order.

• **Something in this skill turned out wrong or missing, or the user says

"file that as an issue"** → §7 (draft an issue for the skill's repository; file only after the user confirms).

4. Using the HPC inside a loop (submit → wait → retrieve → iterate)

Standard loop skeleton:

1. Deploy changed files (full dependency-chain check, see deploy.md), then submit

(see slurm.md).

1. Wait in the background, polling at ≥ 60s intervals (scale up to 5–10 minutes

for long jobs): ``bash ssh "$HPC_HOST" 'squeue --me -h | wc -l' # 0 = everything finished ` When the queue drains, immediately classify with sacct` — COMPLETED vs FAILED vs PREEMPTED. An empty queue does not mean success.

1. Retrieve results (create the local destination first — rsync does not create

a missing parent directory, and then exits 0 having copied nothing): ``bash mkdir -p ./results && rsync -azP "$HPC_HOST":<remote>/results/ ./results/ && ls ./results | wc -l `` Incremental and safe to re-run; the trailing count is the check that files actually arrived — never trust a clean exit alone (see deploy.md, "Transfer commands").

1. Analyze locally → adjust code/parameters → back to 1. Re-run only the missing

tasks using the idempotent submit pattern (slurm.md, "Idempotent submit pattern") — that pattern is what makes the whole loop safely re-entrant.

5. Key cluster facts (quick recall; details in references/)

• New cluster login: login-prod.pax.tufts.edu (load-balanced across

login-p01/02/03). Hostnames without -prod belong to the old, retiring cluster.

• All public partitions cap at 2 days (2-00:00:00). Partitions: batch

(CPU only), gpu (must request --gres), preempt (most resources, but jobs can be preempted and are killed within ~30s — tasks must be idempotent / resumable).

• Home /cluster/home/$USER has a 30GB quota; conda environments and large data

belong in research storage, not home.

• Conda: module load miniforge/25.3.0 (there is no module load python).
• Interactive session QOS: max 4 hours, max 1 GPU.

6. Off-campus access (knowledge, not an operation)

This section is background to explain when the user asks about connection problems — it is not something this skill configures or performs on its own. The skill always just uses $HPC_HOST; whether that route goes through a VPN or a jump host is the user's local SSH client configuration, entirely outside the skill's operational scope.

The login nodes are only reachable from the campus network. From off campus, the official route is the Tufts VPN. A common pain point: the VPN can be slow or flaky, and without it the cluster is unreachable. If the user controls an on-campus machine that is reachable from the internet (e.g. a lab server exposed via ngrok or similar), they can route SSH through it with ProxyJump in their local ~/.ssh/config:

# Jump host: an on-campus machine the user controls
Host my_jump_server
    HostName <server_address>
    User <username>
    Port <port>
 
# HPC via the jump host
Host hpc
    HostName login-prod.pax.tufts.edu
    User <your_utln>
    ProxyJump my_jump_server

After ssh-copy-id to both the jump host and the HPC, ssh hpc is transparent, passwordless, and VPN-free — and everything in this skill works unchanged, since the jump is invisible above the SSH layer. If the user asks for this setup, help them edit their local ssh config; never store the jump host's details anywhere in a repository.

7. Reporting problems with this skill

This skill lives in the public repository <https://github.com/Zhangyanbo/hpc-skills> (directory skills/tufts-hpc/) and is maintained from real usage: when something in it turns out to be wrong or missing, that is worth an issue at <https://github.com/Zhangyanbo/hpc-skills/issues>. Typical triggers, noticed while doing a task:

• a documented command failed or behaved differently from what the skill says

(wrong module name, flag, path, limit);

• a cluster fact here is stale (partition limits, quotas, hostnames, tool

versions);

• a gap that caused an avoidable detour — something you had to discover the

hard way that the skill should have said up front.

Procedure: