The FA5090 Optimization Journey: From 3.23% to 94.6% of Roofline on a Consumer Blackwell GPU
2026-08-12
A seven-version profiling-driven walkthrough of taking a FlashAttention forward kernel on the RTX 5090 (sm_120) from a naive 3.23% to 94.6% of the measured GEMM ceiling, and what each rung of the ladder cost to establish.
gpu-kernel-optimizationflashattentioncudarooflineprofilingsm_120tensor-coreskernel-ladder