huangpustar/metainfer/int8-w8a8-gemm-decode

INT8 W8A8 GEMM — Decode (M <= 32) — gfx928 / SGLang TP4

Worker-29 lineage, DTK 26.04; K500SM_AI/gfx928, 120 CUs, wavefront 64, 64 KiB LDS/CU, 65,536 VGPRs/CU. Validated evidence covers M<=16; M in (16, 32] is a boundary gap — reuse the M=16 recipes below and measure (see §3). This skill is the decode half of the phase split; load int8-w8a8-gemm-foundations for the operator/layout contract, DUMMA and epilogue rules, Graph-safe interface, SGLang integration, and shared guardrails. Baselines are fixed user-supplied Triton CUDA-graph numbers that were never re-measured in this lineage — do not treat any baseline as optimal; "speedup" is always vs that fixed table.

For the four Hy3 TP4 M=16 shapes (2026-08-23, hy3-dsh-tp4-m16-1-7f1fb1d1) the source of truth is the authoritative control-plane fact ledger plus each worker's runs/<shape>/experiments.jsonl — pending worker SKILL.md tables may be stale and must not override the ledger. All four shapes have accepted candidates; no plateau was proven anywhere (`plateau=false`): every accepted number is best-found, not a proven optimum. A shape whose worker timed out / produced no accepted candidate stays unoptimized — never infer success, a speedup, or optimality from a failed or timed-out worker.

Operator: x_q[M,K] int8 @ W[K,N] int8 -> int32 -> * x_scale[M,1] * weight_scale[N,1].T -> bf16[M,N]. Timed region = GEMM + scales + bf16 epilogue + split-K combine inside the CUDA Graph; excludes quantization, weight packing/preprocessing, allocation, JIT, capture. Protocol: unprofiled CUDA-graph replay, 100 warmup / 30 samples x 100 replays, hot cache; report median AND p90; Graph vs Graph only.

1. M = 16 decode (validated)

Use gfx928 INT8 DUMMA:

DUFragment<matrix_a,16,16,32,signed char,row_major>
DUFragment<matrix_b,16,16,32,signed char,row_major>
DUFragment<accumulator,16,16,32,int>

Best validated TP4 dispatch:

The non-split kernel computes a 16x32 output tile with two wavefronts. It loads A/B cooperatively, performs m16n16k32 DUMMA, and writes each accumulator fragment directly from registers through a fused scale/bf16 epilogue.

Split-K requirements:

• Allocate int32 workspace before Graph capture.
• Every launch overwrites every partial tile; no workspace clear is needed.
• Measure partial GEMM plus combine together.
• Do not report only the main kernel as total operator latency.
• Do not restrict split-K to powers of two. Sweep block count near an integer

multiple of the physical CU count, while keeping every K boundary aligned to the DUMMA/staging unit.

For the fully validated M=16,K=4096,N=1536 optimization sequence, byte accounting, ISA evidence, rejected variants, and the 570 GB/s result, read references/wqkv-a-m16-split10-570gbps.md.

1.1 Hy3 TP4 M=16 shapes (authoritative, 2026-08-23)

Each accepted kernel is selected by an exact (m,n,k) shape guard placed before generic tiled/scalar paths so decode, M=3072/4096 prefill, and generic arms stay untouched. HIP_KERNEL_NAME(templated<...>) + hipLaunchKernelGGL are valid on this toolchain and used by every accepted kernel here. Scalar bootstrap kernels (qkv 476.8 us, o_proj 511.4 us, gate_up 331.0 us, down_proj 39.4 us) are correctness-first placeholders, not baselines.

Accepted architectures (reusable recipes):

• qkv (N=2560,K=4096): block-N=64, 256 thr = 4 waves (one 16x16

quadrant/wave), grid = 40 tiles x split-K=6 = 240 blocks = 2 blocks/CU; stage-K=64 double-buffered A+B LDS (10,240 B/block); one 16-B cooperative load/lane/stage; load-bearing tid-linear B lane roles (b_krow = tid>>2, b_nchunk = tid&3 = 16 unique 128-B lines per wave-load); one __syncthreads/stage; non-uniform 64-aligned K slicing ([0,K) exactly once, preserves bit-exact int32 order); fused last-arrival combine tail — monotonic global counters (arrived % SPLIT_K == SPLIT_K-1), __threadfence + atomicAdd, last arriver sums the L2-hot planes ascending, scales, stores one 8-B bf16; counters in the last 160 B of the 16-plane contract workspace, one-time async memset before first launch → one kernel launch per replay. Accepted chain: 476.8 (scalar) → 140.63 → 131.71 → 124.92 → 54.21 → 36.88 → 31.52 → 28.48 → 26.33 → 26.05 us.

• o_proj (N=4096,K=2048): 256 blocks x 128 thr = 2 waves, one 16x16 N

tile/block; in-block split-K=2, 8 uniform stages of K=128 (4 m16n16k32 steps, 8 dwordx2 loads)/wave; register-only K-loop transport with depth-1 prefetch (next stage's 8 dwordx2 loads issued before the current 4-MMAC burst, rotated in after; compiler wait lands at next fill); packed-B fragment-slot layout packed[n_tile][k_step][lane][8] produced once out-of-timed-region for (k,n)==(2048,4096); A stays logical row-major (32 KiB, L2-hot); LDS = two 1 KiB int32 partial planes only; one END-of-K barrier; 44 VGPR/24 SGPR/2048 B LDS, 0 spills.

• gate_up (N=768,K=4096): 16x64 block tile, 2 waves x 2 N-tiles/wave

sharing one A fragment, 12 N-groups; runtime split-K 1..16 default 16 → 192 blocks (1.6/CU, all 120 CUs); whole 256-row split slice staged once (A 16x272, B 64x272; kStagedRowStride=272=256+16 bank skew), batched 16-B/thread loads → ds_write_b128, one barrier, then zero-barrier LDS-only K loop (unroll 2: 2 ds_read2_b32 + 2 ds_read2_b64 + 4 v_mmac); packed [N,K] n-major weight for (k,n)==(4096,768) with col_major B + one 8-B LDS read per fragment; S=16 combine specialization = 96 blocks x 128 thr with all 16 plane loads issued into a register array before the ascending sum; GEMM 123 VGPR/68 SGPR/25,856 B LDS; combine in the timed Graph.

• down_proj (N=4096,K=384): grid = N/16 = 256 blocks x 128 thr = 2 waves,

split-K=2 along K (K-half per wave, 6 unrolled steps); A staged 16x192 row-major, 208-B padded stride (192+16), dwordx4 loads → ds_write_b128; B staged n-major from packed [N][K] transpose P[n*384+k]=W[k*4096+n], 208-B padded column stride, one ds_read_b64 per step (was 8 ds_read_u8 + ~11 VALU); fused per-block LDS combine: wave 0 publishes int32 partial to s_part, one barrier, wave 1 adds s=0 then s=1 (exact int32 order) and emits scaled bf16 from registers; scales prefetched into LDS s_scale[32] at kernel top with stores deferred until after both staging loops (removes prologue vmcnt window); 43 VGPR/18 SGPR/14,464 B LDS.

Cross-shape rules that held (measured):

1. M=16 grid parallelism is scarce — split-K is the occupancy lever.

Sweep non-power-of-two split counts near integer multiples of 120 CUs ({2,3,4,5,6,8,9,10,12,16}); winners were 6 (qkv), 2 (o_proj, down_proj), 16 (gate_up). Keep every K boundary aligned to the staging/DUMMA unit.

1. Direct global fragment loads are poison first: library byte-load +

reassembly + per-load vmcnt before each mmac cost 1.6-3x. Fix: one aligned 8/16-B vector load per lane per fragment/step, via LDS staging (gate_up, down_proj, qkv) or register-only transport (o_proj).

1. Pack the cold once-read weight (B) outside the timed region/Graph

into the fragment's layout ([N,K] n-major transpose, per-tile/ fragment-slot packs) so each fragment is one contiguous dwordx2/b64 read; leave small L2-hot A logical. Packing alone was small; it is the prerequisite for vectorized B loads.

1. Zero in-loop barriers for decode; one END-of-K barrier for the LDS

int32 partial-plane combine is cheap and order-independent (bit-identical).

1. Split-K combine is part of the operator wall — never time the partial

kernel alone. Fuse it where it wins (qkv last-arrival tail; down_proj per-block LDS) but not universally: gate_up's atomic-accumulator + finalize fusion regressed 43.8%.

1. LDS bank skew: pad row/column strides to 16-B-aligned non-power-of-two

values (272=256+16, 208=192+16, 80/72/68/144 elsewhere); strides ≡ 0 (mod 128 B) alias every row onto one bank phase.

1. **Exact int32 accumulation order (k-ascending per split, ascending split

sum) is preserved across all accepted variants → outputs are bit-identical (0 mismatches).**

1. Launch geometry before micro-opt (down_proj: 256x1-wave 68.9 us →

128x2-wave 15.4 us); verify the exact code object of the timed run (digest-matched), not a stale sibling; source-level prefetch is not overlap without ISA proof (global_load_dwordx4 -> v_mmac -> vmcnt(0) -> ds_write_b128).

1.2 Fallback routing (must stay byte-identical and building)

• qkv: FUSED=false partial + quadrant combine kernel for workspaces < 16

planes; generic scalar fallback for every other (m,n,k) including paired M=2.

• o_proj: scalar fallback decodes the packed-B layout for (k,n)==(2048,4096);

M=2 fallback exact.

• gate_up: scalar fallback decodes the [N,K] pack for (k,n)==(4096,768);

identity copy otherwise; generic single-buffer tiled arms kept untuned.

• down_proj: unsplit two-wave DUMMA (grid 128) for other M=16 shapes; generic

scalar with a b_transposed flag for M=2.

• Routing rule: exact-shape guard → tuned kernel; any other shape → generic

tiled path or scalar fallback, never the tuned kernel.

2. 1 <= M < 16

The implemented compatibility path zero-pads A to 16 rows in LDS and masks output rows. It is correct and Graph-safe, but performance must be measured for each M. Do not assume the M=16 dispatch is optimal for M=1/2/4/8.

For the validated M=2,K=4096,N=1536 split-K=8 route, ISA/PMC evidence, bandwidth accounting, failed B-staging/prefetch/finalize variants, the other M=2 baselines, and acceptance gates, read references/m2-decode-search.md. Do not route other M=2 shapes to a custom kernel until that reference's exactness and repeated Graph speedup gates pass; retain Triton otherwise.

3. Boundary: 16 < M <= 32

No accepted candidate has been measured in (16, 32]. Reuse the M=16 recipe (split-K + fused combine family) and measure per exact M before committing a route. Keep a decode-compatible arm and a staged prefill arm for the same (K,N) until measured; the decode direct-B-from-global pattern does not transfer to larger M, and prefill's no-split-K rule does not apply here.

4. Validated evidence

M=2 decode, 2026-08-03

Fair preallocated-output baseline, hot cache, 50 warmups, 50 samples, 20 launches/sample; exact output check passed: