Add tuning notes for Hopper 16-bit dense GEMM#2
Open
hamuzhan wants to merge 1 commit into
Open
Conversation
Document kernel-selection guidance for FP16/BF16 dense GEMM on SM90, based on profiling on H100: - accumulator register budget per kernel schedule, and the local-memory spilling that happens for oversized per-warp-group tiles - small-M (skinny/decode) configs and the instantiation-level setting to generate them - rasterization and swizzle guidance for memory-bandwidth-heavy shapes - sustained-clock effects on performance measurement Signed-off-by: hamuzhan <hamzayigitkltr@gmail.com>
1100001 to
243dcb6
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Add tuning notes for Hopper 16-bit dense GEMM
Summary
Adds a "Practical tuning notes for Hopper 16-bit dense GEMM" subsection to
media/docs/cpp/efficient_gemm.md, at the end of the existing "Hopper WarpSpecialization" section.
The notes cover four points for selecting FP16/BF16 dense GEMM kernels on SM90,
based on profiling on H100 (CUDA 13.2) with the CUTLASS profiler and Nsight
Compute:
formula for warpspecialized, ping-pong, and cooperative schedules, and the
local-memory spilling that happens for per-warp-group tiles larger than
64x256. This matches the pruning in the companion PR.
M <= 128, which tile/cluster/schedule shapes recover throughput (measured up
to +44% at M=16 and up to +74% at M=128, FP16 and BF16), and the
CUTLASS_LIBRARY_INSTANTIATION_LEVELsetting to generate them.--raster_order=along_m --swizzle_size=2on a DRAM-bound shape (median ofinterleaved runs), and a note that
along_ncosts about 25% on very-large-Nshapes.
board power limit.
Motivation
These are common performance issues for users selecting SM90 kernels by hand
rather than through autotuning. The existing section describes the kernel
designs but not the selection limits, so the notes go next to that description.
No code changes. Documentation only.