production· client work
CUDA kernels for NVIDIA Blackwell
Hand-written CUDA kernels for SM120 covering fp16 GEMM, GQA attention with M-RoPE, LayerNorm, and fused UNet upsampling.
SM120
Problem
New GPU architectures ship before tuned kernels for every model shape exist. Getting real speed out of Blackwell (SM120) for transformer and diffusion workloads meant writing kernels by hand.
Approach
- fp16 GEMM. Tiled matrix multiply using tensor cores, with shared memory staging and tuned tile shapes for the target sizes.
- GQA attention with M-RoPE. Grouped-query attention with multimodal rotary position embedding applied inside the kernel, so Q and K are never written back in rotated form.
- LayerNorm. Single-pass, warp-level reductions with vectorized loads.
- Fused UNet upsampling. Upsample, convolution input preparation, and elementwise ops fused into one kernel to cut memory round trips.
Each kernel was validated against a PyTorch reference for correctness, then profiled with Nsight Compute to close the gap to hardware limits.
Results
Measurable speedups over the baseline implementations on the target shapes. Specific numbers and code are under contract.
Stack
CUDA C++, tensor cores, Nsight Compute, PyTorch (reference and tests).
Contract work. No client names or code are published.