ai-modcon/prism/prism-scaling-and-isoflop

Claude Code and agent skill by ai-modcon/prism

Trust tier 3
  • community
Source check pending

Checking that the SKILL.md file is still in the repository.

Measure PRISM throughput and run scaling / IsoFLOP studies. Use when benchmarking samp/s, running the per-modality sweep, executing the IsoFLOP pipeline (calibrate → plan → launch → collect → fit → plot), or reasoning about MFU / scaling efficiency. Triggers: "throughput sweep", "isoflop", "samp/s", "scaling efficiency", "MFU", "benchmark_throughput", "run_sweep", "compile storm", "N_opt / compute-optimal", "calibration json".

Install

npx vskill@latest install ai-modcon/prism/prism-scaling-and-isoflop

SKILL.md

PRISM — Scaling & IsoFLOP

Two related workflows: measuring throughput (per-modality sweep, benchmarks) and compute-optimal scaling (the IsoFLOP pipeline). Before trusting any scaling exponent, confirm the data pipeline is honest (prism-data-pipeline / ml-data-pipeline-correctness) and that the run wasn't hit by the intermittent backward collapse (prism-distributed-strategy).

Throughput measurement

ToolPurpose
tools/benchmark_throughput.pyMicro-benchmarks: --test {forward_only,batch_scaling,fwd_bwd,sync_overhead,ddp_overhead,all}, --model {1b,7b}, --seq-len
tools/run_sweep.pyPer-modality sweep harness (--preset text_image,text_ts,…, --designs PRISM-MODALITY-SMOKE-1N, --storage {lustre,daos}, --sweep-id)
tools/perf_aggregate.pyScan outputs/ for perf.jsonl → CSV (--filter key=value, --per-modality)

The per-modality sweep (PRISM-MODALITY-SMOKE-1N, presets in experiments/modality_presets.yaml) runs 5 non-VLA cells + a VLA cell. Watch for the WEBDATASET_LOCAL_PATH leak (one cell's shards bleeding into another) — fixed in PR #73 but easy to reintroduce; verify per-cell shard paths.

IsoFLOP pipeline (tools/isoflop_*.py)

Run in order:

StepToolKey args
1. Calibrateisoflop_calibrate.py--backbone, --projector-variant {BASE,W2X,W4X,D2X,D4X}, --family {text_image,text_ts,text_graph}
2. Planisoflop_plan.py--family, --budgets <flops,…>, --scaling-study-dir, --calibration-dir
3. Launchisoflop_launch.py--plan <yaml>, --launcher tools/launch_aurora_daos.py, --csv
4. Collectisoflop_collect.py--outputs <root>, --csv, --eval-window 5, --force
5. Fitisoflop_fit.py--csv, --family, --backbone, --regime {projector_only,encoder_projector,e2e,any}, --bootstrap 1000, --output-json/md
6. Plotisoflop_plot.py--csv, --family, --regime, --outdir

Calibration JSONs live in scaling-study/calibration/. The projector variant ladder (BASE/W2X/W4X/D2X/D4X = hidden_mult/num_layers) is the axis that spreads N; see prism-adding-a-modality for where it's defined.

Hard-won scaling facts (don't re-derive)

  • PRISM runs at ~6% MFU (vs AGPT-2B's ~27.6%). The bottleneck is low

arithmetic intensity (~708 tokens/sample), not scaling efficiency. There's 4–5× single-node headroom before multi-node scaling even matters — fix intensity first.

  • Full-shard vs HSDP: for our comm-bound 0.6B/seq-512 regime, HSDP

shard_grad_op beats FSDP full_shard by ~42–47%. AGPT's 94% full_shard number does not transfer.

  • 10N throughput plateaus. Follow-up matrices (E15–E20b) falsified the

"topology lifts throughput" hypothesis; the 20N mechanism is CPU dispatch, not AllReduce. Production launcher config is unchanged; only MNIC=global+cxi naming gave +1.4%.

  • Compile storm is scale-dependent (10N+, not 2N); static==dynamic; not

caused by gradient checkpointing (inert at GCF=1).

  • torch 2.13 vs 2.10: wins at ≤4N (1N +37%, 2N +33%, 4N +14%), ~break-even at

10N (gain decays with N per Amdahl, compute-path only).

  • Round-1 IsoFLOP fit was underdetermined by N-spread (structural, not

noise): BASE@3e17 stdev≈0.012, between-variant signal ~11× noise. Need 7B+32B backbones to spread N.

See also

`docs/results/per_modality_sweep.md`.

Security

  • Security scanPassed, 90 of 100Pattern checks on the skill's files.
  • Model reviewPassed

See the full security report

Quality evaluation

  • Task evaluationNot runNo evaluation yet. Run one in Skill Studio.

See evaluation results

Source

Popularity

Repository stars
12
Repository forks
1
7-day trend score
0

Stars and forks are for the whole GitHub repository, not this skill alone. Not refreshed yet.

Works with

Any agent vskill installs to: 56 targets in all. These 39 have a page on this site.

  • Claude Code
  • Cursor
  • GitHub Copilot
  • Windsurf
  • Codex
  • Gemini CLI
  • Cline
  • Amp
  • Roo Code
  • Goose
  • Continue
  • Kilo Code
  • Junie
  • Trae
  • OpenHands
  • Qwen Code
  • AI Agent
  • Tabnine Chat
  • Sourcegraph Cody
  • Amazon Q
  • Firebase Genkit
  • Sweep
  • Devin
  • Phind
  • Mentat
  • Claude Desktop
  • Aider
  • GPT Pilot
  • Plandex
  • bolt.new
  • v0
  • Replit Agent
  • Anse
  • TypingMind
  • BetterChatGPT
  • LibreChat
  • Warp AI
  • Pieces
  • Fig AI
Embed badge
vskill VERIFIED badge
[![vskill VERIFIED](https://verified-skill.com/api/v1/skills/ai-modcon/prism/prism-scaling-and-isoflop/badge)](https://verified-skill.com/skills/ai-modcon/prism/prism-scaling-and-isoflop)

Something wrong with this skill or its page? Tell us.