Skip to content

Add tuning notes for Hopper 16-bit dense GEMM#2

Open
hamuzhan wants to merge 1 commit into
mainfrom
sm90-16b-tuning-notes
Open

Add tuning notes for Hopper 16-bit dense GEMM#2
hamuzhan wants to merge 1 commit into
mainfrom
sm90-16b-tuning-notes

Conversation

@hamuzhan

@hamuzhan hamuzhan commented Jul 7, 2026

Copy link
Copy Markdown
Owner

Add tuning notes for Hopper 16-bit dense GEMM

Summary

Adds a "Practical tuning notes for Hopper 16-bit dense GEMM" subsection to
media/docs/cpp/efficient_gemm.md, at the end of the existing "Hopper Warp
Specialization" section.

The notes cover four points for selecting FP16/BF16 dense GEMM kernels on SM90,
based on profiling on H100 (CUDA 13.2) with the CUTLASS profiler and Nsight
Compute:

  1. Accumulator register budget per schedule. The per-warp-group accumulator
    formula for warpspecialized, ping-pong, and cooperative schedules, and the
    local-memory spilling that happens for per-warp-group tiles larger than
    64x256. This matches the pruning in the companion PR.
  2. Small-M (skinny/decode) problems. Why wide default tiles leave SMs idle at
    M <= 128, which tile/cluster/schedule shapes recover throughput (measured up
    to +44% at M=16 and up to +74% at M=128, FP16 and BF16), and the
    CUTLASS_LIBRARY_INSTANTIATION_LEVEL setting to generate them.
  3. Rasterization and swizzle for memory-heavy shapes. Measured +8-10% from
    --raster_order=along_m --swizzle_size=2 on a DRAM-bound shape (median of
    interleaved runs), and a note that along_n costs about 25% on very-large-N
    shapes.
  4. Sustained-clock effects. A measurement note for near-peak kernels under the
    board power limit.

Motivation

These are common performance issues for users selecting SM90 kernels by hand
rather than through autotuning. The existing section describes the kernel
designs but not the selection limits, so the notes go next to that description.

No code changes. Documentation only.

Document kernel-selection guidance for FP16/BF16 dense GEMM on SM90,
based on profiling on H100:
- accumulator register budget per kernel schedule, and the local-memory
  spilling that happens for oversized per-warp-group tiles
- small-M (skinny/decode) configs and the instantiation-level setting
  to generate them
- rasterization and swizzle guidance for memory-bandwidth-heavy shapes
- sustained-clock effects on performance measurement

Signed-off-by: hamuzhan <hamzayigitkltr@gmail.com>
@hamuzhan hamuzhan force-pushed the sm90-16b-tuning-notes branch from 1100001 to 243dcb6 Compare July 7, 2026 10:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant