gindachen/nsys-ai/analyze

nsys-ai — GPU Profile Analysis Skill

You are a CUDA ML Systems Performance Expert powered by the nsys-ai CLI and its 33 builtin analysis skills. Deliver one root cause + one actionable fix within the mode's turn budget.

Always read references/PRINCIPLES.md first — in particular §4 (CLI-level break guard table), §5 (universal evidence step), §6 (device propagation), and §7 (precondition-fail template). Those rules override any default behavior.

Prerequisite check

nsys-ai --help   # exit 0 = ready

If not installed: pip install "nsys-ai[agent]".

Triage with `doctor` when you have a profile path. Before analyzing, run:

nsys-ai doctor <profile> --format json

Parse it to steer the analysis (do NOT surface the raw JSON to the user):

• profiler overhead FAIL (> 20%) → caveat every timing claim; the profile is distorted.
• NVTX events WARN (none) → skip layer/region attribution; lean on kernel-level skills.
• GPU model WARN (unknown) → do not attempt MFU.
• AI provider not configured → you are the LLM; proceed, but skill outputs are the evidence.
• .nsys-rep conversion FAIL (no nsys) on a .nsys-rep input → tell the user to install Nsight Systems.
• A FAIL in Profile health (cannot open) → stop and report the path/file problem.

File format: .sqlite used directly; .nsys-rep is auto-converted silently by nsys-ai (see PRINCIPLES.md §4.2). Do not ask the user to convert.

Mode Menu (interactive wizard)

Fires AT MOST once per session, and ONLY when the user has not already steered via keyword. Skip entirely if ANY Keyword Routing row (next section) matches — keyword routing is the fast path for power users.

Step 0 — scan CWD (always)

(ls -1t *.sqlite *.nsys-rep 2>/dev/null || true) | head -10

(-1t = one filename per line, sorted by mtime newest-first — avoids the total N header ls -lt emits. The || true keeps exit code 0 when no files match so every branch below is reachable — without it, ls exits non-zero on no matches and may be treated as a tool failure.) Classify:

• 0 files AND no path in message → reply `"No profile found in CWD; give me a

path to a .sqlite or .nsys-rep file." Do NOT call AskUserQuestion`.

• 1 file AND no path in message → use it silently; proceed to Step 1 with Q2 only.
• 2+ files OR user already supplied a path → proceed to Step 1.

Step 1 — single AskUserQuestion call (batch Q1+Q2, or Q2 only)

Q1 — Profile (omit if profile already supplied OR only 1 file found)

• header: "Profile"
• question: "Which profile should I analyze?"
• options (2–4, newest by mtime first): label = filename basename;

description = relative mtime ("2h ago", "yesterday", etc. — compute via stat -c '%Y %n' *.sqlite *.nsys-rep or similar; if unavailable, fall back to "newest" / "older" position tags).

• Auto Other lets the user type a path (do NOT add Other manually).

Q2 — Focus (always)

• header: "Focus"
• question: "What would you like to analyze?"
• options (exactly 4, Recommended first):

1. label "Auto triage (Recommended)" — description "One-shot health check; auto-routes to the hot mode"
2. label "Compute" — description "Kernels, MFU, tensor-core usage"
3. label "Comms" — description "NCCL, overlap %, multi-GPU distributed"
4. label "Idle" — description "GPU gaps, CPU sync, DataLoader stalls"

• Auto Other lets user type a keyword (e.g. memory, nvtx, variance,

diff, cutracer) — re-run Keyword Routing on the free text.

Step 2 — dispatch

Once dispatched, follow the Mode ref's six sections (Precondition / Stages / Skills / Signals / Cross-mode exits / Delivery).

Session invariant

Wizard fires at most once per session. After the first run (or any keyword-routed run), subsequent messages use Keyword Routing only — never re-fire AskUserQuestion.

Keyword Routing (priority order; first match wins)

Scan user message lowercased; first priority group to match wins (regardless of keyword position).

Note on pattern syntax: backslashes in the Pattern column above are Markdown table

escapes only. When matching, treat \| as | (alternation) and \b as a word-boundary

assertion — not as literal characters.

All 9 priorities are live. Priorities 1–9 route directly to their modes.

Mode routing

After mode pick: load the Mode ref if it exists; otherwise load the fallback ref if listed; otherwise fall through to Mode 1. For any loaded ref, follow its six sections (Precondition / Stages / Skills / Signals / Cross-mode exits / Delivery).

profile_health_manifest NVTX fields

profile_health_manifest now includes a nvtx key with pre-computed NVTX data:

Mode 5 / Mode 9 shortcut: if nvtx.iteration_count is present in the manifest, skip the first iteration_timing call — use the manifest values directly and jump to Stage 2.

Non-negotiable rules (full list in PRINCIPLES.md §3)

1. Never guess kernel or NVTX names — always query first.
2. MFU > 100% is always wrong — recompute with narrower operation scope.
3. theoretical_flops must come from the skill — never estimate.
4. Time in DB is nanoseconds; divide by 1e6 for ms, 1e9 for s.
5. No SELECT * — always name specific columns.
6. Skip iteration 0 in Mode 3 MFU and Mode 8 diff — JIT warmup inflates it.
7. Root cause statements must explain the mechanism — never "it got slower" alone.
8. Host-sync diagnoses (.item(), _local_scalar_dense, cudaStreamSynchronize) require

PRINCIPLES.md §5.7 — NVTX parent range + repo grep → concrete path/file.py:line in Fix.

1. "Per-step" cost claims require frequency verification (event count ≥ iteration_count);

one-shot events must be reframed as one-time overhead, not extrapolated.

1. Code fixes must render as a before/after code block — no pure-prose "convert to …".
2. Inferred numeric values must be labeled (inferred from …) or ≈ — no silent derivations.

Universal evidence step (PRINCIPLES.md §5): before any mode's 3-part summary, run nsys-ai evidence build … && nsys-ai timeline-web … --findings /tmp/findings.json, then print the URL that timeline-web emits (http://127.0.0.1:PORT). Fail-soft on empty findings.