ascend-ai-coding/awesome-ascend-skills/ascend-profiling-anomaly

Ascend Profiling Anomaly Discovery Skill

Purpose

Analyze Ascend NPU profiling data through three parallel pipelines:

1. Structure breakdown: step → structure (layer) → block / side → op → PMU judgement — answers where the time goes.
2. Anomaly discovery: step → device busy union → bubble detection → anomaly tags → soft attribution — answers what looks unnatural and where hidden issues may lurk.
3. Model architecture analysis: FIA timeline → pass boundaries → layer classification → per-layer sub-structure → communication pipeline → architecture summary — answers what is this model and how does each component execute. Produces a separate Markdown report file.

The core philosophy is separation of concerns: "anomaly exists" is a hard fact derived from device intervals; "why it exists" is a soft attribution that may require additional evidence. Even under weak profiling configurations (no stacks, no shapes, sparse host events), the skill must still reliably surface device idle bubbles and risk labels.

Reference Files — When to Read

Read these before starting analysis:

Pipeline Overview

The full state machine:

INGEST → INVENTORY → FACT_EXTRACTION → CANDIDATE_STEP_DETECTION →
STEP_GROUPING → MACRO_STEP_RESOLUTION → SEGMENTATION →
CLOCK_ACCOUNTING → ANOMALY_DISCOVERY → PERF_JUDGEMENT →
RECOMMENDATION → RENDER → DONE
                                        ↘
                          ARCHITECTURE_ANALYSIS → ARCH_REPORT_RENDER → ARCH_REPORT_SAVE

ANOMALY_DISCOVERY sits after CLOCK_ACCOUNTING and before PERF_JUDGEMENT. It receives already-segmented steps and structures, and runs the bubble detection pipeline on top of them.

ARCHITECTURE_ANALYSIS runs in parallel with PERF_JUDGEMENT, using the same segmented data from CLOCK_ACCOUNTING plus FIA timeline analysis. It produces a separate Markdown report file saved alongside the profiling data.

Data Hierarchy: Step → Structure → Block/Side → Op

Understanding how raw kernel data maps to each level is essential. Read references/kernel_data_guide.md for the full column schemas and parsing details. Here is the conceptual overview:

Level 0: Raw Kernels

Each row in kernel_details.csv represents a single device kernel execution — one invocation of an AI Core task, an AI CPU task, or an HCCL communication task. Key fields: Name, Task Type, Start Time(us), Duration(us), Wait Time(us), Accelerator Core, Stream ID, Input Shapes, Output Shapes.

Level 1: Step

A step is one training/inference iteration. Steps are identified by ProfilerStep#N user annotations or Iteration#N markers in trace_view.json. Each step defines a service window [S_i, S_{i+1}). All kernels whose start time falls within this window belong to step i.

At step level, compute:

• Total service time, device busy union, underfeed ratio
• Prelaunch gap, tail gap, internal bubbles
• Per-step anomaly tags and soft root-cause labels

Level 2: Structure (Layer)

Within a step, kernels form repeating structures — typically corresponding to model layers (e.g., transformer blocks, attention layers, MLP blocks). Segmentation identifies these by:

1. Repeating name-pattern sequences in the kernel timeline
2. Significant time gaps between kernel groups
3. User annotations marking layer boundaries (if present)

Each structure contains a contiguous span of kernels within the step window.

At structure level, compute:

• Structure wall time, device busy union within structure span
• Structure share of total step time
• Per-structure bubble metrics (is the bubble inside this structure or between structures?)

Level 3: Block / Side

Within each structure, kernels split into:

• Block (main compute path): The dominant chain of AI Core kernels forming the forward or backward pass of this layer. These execute on the main compute stream.
• Side (auxiliary ops): Everything else — small element-wise ops, communication (HCCL), AI CPU fallback ops, memory copies, synchronization events. These may execute on separate streams.

At block/side level, maintain four timing perspectives simultaneously:

• wall_ms — wall-clock time from first kernel start to last kernel end
• busy_union_ms — merged device-busy time (accounts for multi-stream overlap)
• kernel_sum_ms — arithmetic sum of all kernel durations (ignores overlap)
• total_cost_ms — sum of duration + wait for all kernels

Conclusions based on only one metric are incomplete. A block appearing heavy in kernel_sum but light in wall means high stream parallelism. A side appearing heavy in total_cost but light in duration means wait-anchor false hotspot risk.

Level 4: Op (individual operator)

The finest grain. Each op may produce one or many device kernels. Op-level analysis handles:

• Top ops by total cost vs. top ops by kernel duration (these rankings often differ)
• Wait-anchor detection: ops with wait_ratio > 0.95 and tiny duration but high total_cost
• AICPU classification: ops running on AI CPU instead of AI Core, classified by masked_ratio
• Small-op initial judgement: whether small individual ops are real inefficiencies or noise

Anomaly Discovery Pipeline (Detail)

Phase 1: BUILD_DEVICE_INTERVALS

For each step, collect all device kernel intervals from kernel_details.csv:

device_intervals = []
for each kernel row where Start_Time_us is within step window:
    s = max(step_start_us, row.Start_Time_us)
    e = min(step_end_us, row.Start_Time_us + row.Duration_us)
    if e > s:
        device_intervals.append(Interval(s, e))

Key rules:

• Clip kernel intervals to the step window boundary
• Apply communication dedup rules BEFORE interval statistics (see references/kernel_data_guide.md section on comm dedup)
• Include AI_CORE, AI_CPU, and HCCL tasks — all count as "device busy"
• When multiple streams exist, intervals from all streams are collected into the same set

Phase 2: MERGE_INTERVALS

Sort intervals by start time, merge overlapping ones:

merged = merge(device_intervals)  # see scripts/reference_host_gap_branch.py
busy_union = sum of merged segment durations

This produces the merged busy segments from which all bubble metrics derive.

Phase 3: GAP_CLASSIFICATION

From the merged segments, compute per-step metrics:

• service_ms = step window duration
• device_busy_union_ms = sum of merged segment durations
• underfeed_ms = service − busy_union
• underfeed_ratio = underfeed / service
• prelaunch_gap_ms = start(first_merged_segment) − step_start
• tail_gap_ms = step_end − end(last_merged_segment)
• internal_bubble_total_ms = sum of gaps between consecutive merged segments
• largest_internal_bubble_ms = max gap between consecutive merged segments
• bubble_count = number of inter-segment gaps

Phase 4: HOST_EVIDENCE_COLLECTION

For each bubble window (gap between merged segments, or prelaunch/tail gap), scan the same time range in host events from trace_view.json:

Host event categories to collect:

• cpu_op / python_function / user_annotation — general host activity
• AscendCL@* — ACL runtime events
• HostToDevice / torch_to_npu / aclrtMemcpy* / aclrtSynchronize* — sync/copy markers
• c10d / Hccl / hcom / StreamWaitEvent / Notify_Wait — communication markers

Compute overlap ratios:

• host_visible_coverage_ratio = fraction of bubble covered by any host event
• sync_marker_overlap_ratio = fraction covered by sync/copy markers
• comm_marker_overlap_ratio = fraction covered by communication markers

Phase 5: SOFT_ATTRIBUTION

For each significant bubble, assign probability-level labels based on overlap ratios:

Multiple labels can co-exist. These are explicitly NOT unique root causes.

Phase 6: ANOMALY_TAGGING

Apply the anomaly tags from references/rulebook.md decision tables. Core tags:

Bubble severity: DEVICE_IDLE_GAP_HEAVY, PRELAUNCH_GAP_HEAVY, TAIL_GAP_HEAVY, INTERNAL_BUBBLE_HEAVY

Risk tags: HOST_ORIGINATED_RISK, COMM_SYNC_RISK, WAIT_POLLUTION_RISK, WAIT_ANCHOR_FALSE_HOTSPOT, AICPU_EXPOSED_RISK, UNTRACED_HOST_BLOCKING_RISK, PARTIAL_CAPTURE_BOUNDARY, VARIABLE_SHAPE_SAME_TEMPLATE

Phase 7: WAIT_ANCHOR_SCAN

At op level, scan for false hotspots:

wait_ratio = wait_us / (duration_us + wait_us)
if wait_ratio > 0.95 and duration_us < 10.0 and total_cost_rank <= 10:
    tag WAIT_ANCHOR_FALSE_HOTSPOT

These ops absorb idle wait time and appear expensive, but their kernel execution is negligible. Demote them in root-cause ranking.

Phase 8: GROUP_AGGREGATION

Aggregate step-level metrics by step_group_id:

• Compute avg, median, P90, P95 for each bubble metric
• recurring_bubble_pattern = true if ≥60% of steps in group have bubble_count > 0
• dominant_idle_pattern = whichever of prelaunch/internal_bubble/tail contributes most

Phase 9: PRODUCE_OUTPUT

Merge anomaly results with structure breakdown. The report MUST include:

Hidden Issue Discovery section

• Dominant step/group bubble metrics (service, busy_union, underfeed_ratio)
• Raw kernel evidence table: for each top bubble window, list the kernel(s) immediately before and after the gap — their names, task types, durations, streams — so the human expert can locate the exact spot in the timeline
• Top 5 bubble windows with timestamps, scope, and host evidence
• Bubble periodicity statistics across the step group
• Host evidence coverage assessment
• Soft root-cause labels with evidence chains
• Follow-up sampling recommendations

Bubble-first Summary (fixed 5-question template)

1. Are there significant device idle bubbles?
2. Which step type/group do they concentrate in?
3. Are they primarily prelaunch / tail / internal / inter-step?
4. Is there significant host-originated risk?
5. Is evidence sufficient for root cause? If not, say so explicitly.

Structure-Level Bubble Drill-Down

For the dominant step, break down bubble contributions per structure:

• Which structure(s) contain the largest internal bubbles?
• Are bubbles concentrated at structure boundaries (between layers) or within structures?
• For structures with large bubbles, what are the surrounding kernel names and task types?

Model Architecture Analysis Pipeline (Separate Markdown Report)