Mog9/FlashAttention-CuPy

Flash Attention from scratch, tiled CUDA forward kernel, online softmax with running max and correction factor, recomputation trick in backward, O(N) memory, full forward and backward verified against PyTorch autograd to 1e-6.

语言：Python · GitHub 星标：19 · Fork：1

在 GitHub 上查看原仓库