Block-wise attention forward pass without the full score matrix
Implement scaled dot-product attention that processes queries and keys in blocks with a running softmax normalization, so the quadratic score matrix is never materialized. Reported on 1point3acres' Hugging Face bank; reconstructed in our own words.
No answer yet. Write the first one; the best answer gets endorsed.