alterlab-ieu/alterlab-academic-skills/alterlab-remote-compute

Remote Compute

Overview

Foundation-model workloads (protein folding, backbone diffusion, single-cell models) need a GPU and can run for minutes to hours — too long to sit in a synchronous call. This skill is the provider-agnostic dispatch layer the GPU skills build on: a single submit → poll → harvest contract that works the same whether the backend is a SLURM cluster, Modal, RunPod, or GCP. You describe the job once; the dispatcher submits it, returns a handle, polls status to a terminal state, and harvests the output artifacts.

It does not wrap any single model — each model skill (alterlab-alphafold, alterlab-boltz, alterlab-proteinmpnn, …) describes what to run; this skill describes where and how to run it.

When to Use This Skill

Use this skill when the user wants to:

• Submit a batch job to a SLURM/HPC cluster and track it to completion (sbatch → sacct).
• Run a GPU job on a managed provider (Modal, RunPod, GCP Batch / Vertex AI) and retrieve results.
• Write a portable job wrapper that runs the same payload across more than one backend.
• Poll a long-running remote job's status and harvest its output files/artifacts.

Does NOT Trigger

The submit → poll → harvest contract

Every backend implements three verbs. Keeping the payload backend-independent is what makes a model skill portable across an HPC allocation and a cloud GPU:

1. submit(spec) → handle — enqueue the job; return an opaque handle (SLURM job id, Modal

call id, RunPod job id, GCP operation name).

1. poll(handle) → status — map the backend's states to a common vocabulary:

PENDING | RUNNING | SUCCEEDED | FAILED | CANCELLED | UNKNOWN. Poll on a backoff; never busy-loop.

1. harvest(handle) → artifacts — copy the declared output files back to a local out/

directory (scp/rsync from HPC scratch; object-store download for cloud).

scripts/dispatch.py implements submit and poll (plus cancel for SLURM) for the SLURM and a generic REST backend, and defines the status vocabulary so model skills can depend on it; harvest is a copy step (rsync/scp or object-store download) shown in references/providers.md. Provider-specific command and API detail lives in references/providers.md (loaded on demand).

Core Capabilities

1. SLURM / HPC clusters

The classic scheduler path. Submit a job script with resource directives, then poll with sacct (authoritative for terminal state) rather than only squeue (which drops finished jobs from its default view):

# Submit; capture the numeric job id from "Submitted batch job <id>"
JOBID=$(sbatch --parsable run.slurm)
 
# Poll to a terminal state (COMPLETED / FAILED / CANCELLED / TIMEOUT)
sacct -j "$JOBID" --format=State,ExitCode,Elapsed --noheader --parsable2

A minimal GPU job script (run.slurm) — one GPU, an 8-hour wall clock, results on scratch:

#!/bin/bash
#SBATCH --job-name=fold
#SBATCH --gres=gpu:1
#SBATCH --time=08:00:00
#SBATCH --output=%x-%j.out
srun uv run python run_model.py --in input.fasta --out "$SCRATCH/out"

Full directive reference, array jobs, and squeue/scancel usage: see references/providers.md.

2. Modal — managed serverless GPU

For serverless containers, autoscaling, and .remote() dispatch, this skill defers to alterlab-modal, which owns the Modal SDK surface. Use Modal when you want zero cluster management and per-second GPU billing. This skill's role is only to treat a Modal call as one submit → poll → harvest backend when a workflow needs to stay provider-agnostic.

3. RunPod — on-demand GPU pods

RunPod exposes GPU pods and a serverless endpoint API. Submit to a serverless endpoint and poll the returned job id; the API key is read from RUNPOD_API_KEY and sent as a bearer token. Endpoint/run/status paths and the pod vs. serverless trade-off are in references/providers.md.

4. GCP — Batch and Vertex AI

Google Cloud offers Batch (containerized batch jobs with GPU allocation) and Vertex AI custom jobs (ML-oriented, managed). Both follow submit → poll (operation/job state) → harvest (read outputs from a GCS bucket). Auth uses Application Default Credentials; the target bucket comes from an env var. Command/SDK detail: references/providers.md.

5. Portable job wrapper

scripts/dispatch.py is a stdlib-only CLI that runs the same job spec across backends:

# Submit a SLURM job and print the handle
python scripts/dispatch.py submit --backend slurm --script run.slurm
 
# Poll a handle to a normalized status
python scripts/dispatch.py poll --backend slurm --handle 123456
 
# Generic REST backend (RunPod-style): endpoint + key from env
python scripts/dispatch.py submit --backend rest \
  --endpoint "$RUNPOD_ENDPOINT" --payload spec.json

It shells out to sbatch/sacct for SLURM and uses urllib for the REST backend — no third-party dependencies, so it runs anywhere Python does.

Validation and status semantics

• Trust the accounting record, not the queue. On SLURM, a job missing from squeue may

have finished or failed — resolve terminal state with sacct/exit code, not absence.

• Poll with backoff (e.g. 10s → 30s → 60s, capped) so you neither hammer the scheduler

nor miss a fast job.

• Always check the exit code, not just the state string; a job can report COMPLETED

while the payload wrote no artifacts. Harvest, then verify the expected files exist.

• Never hardcode credentials. Read RUNPOD_API_KEY, GCP ADC, and cluster hosts from the

environment; the dispatcher refuses to run if a required secret is unset.

Resources

• references/providers.md — per-backend command/API detail (SLURM directives, RunPod

endpoints, GCP Batch/Vertex, Modal cross-link) loaded on demand.

• scripts/dispatch.py — stdlib-only submit/poll/cancel CLI for SLURM + generic REST.

Part of the AlterLab Academic Skills suite.