Resultado de búsqueda
27 de may. de 2022 · View PDF Abstract: Transformers are slow and memory-hungry on long sequences, since the time and memory complexity of self-attention are quadratic in sequence length. Approximate attention methods have attempted to address this problem by trading off model quality to reduce the compute complexity, but often do not achieve wall-clock ...
We argue that a missing principle is making attention algorithms IO-aware— accounting for reads and writes between levels of GPU memory. We propose FLASHATTENTION, an IO-aware exact attention algorithm that uses tiling to reduce the number of memory reads/writes between GPU high bandwidth memory (HBM) and GPU on-chip SRAM.
We propose FlashAttention, a new attention algorithm that computes exact attention with far fewer memory accesses. Our main goal is to avoid reading and writing the attention matrix to and from HBM. This requires (i) computing the softmax reduction without access to the whole input (ii) not storing the large intermediate attention matrix for ...
27 de may. de 2022 · TLDR. This work proposes FlashAttention, an IO-aware exact attention algorithm that uses tiling to reduce the number of memory reads/writes between GPU high bandwidth memory (HBM) and GPU on-chip SRAM, and is optimal for a range of SRAM sizes. Expand. [PDF] Semantic Reader.
Attention mechanism. Block-oriented device. Asymmetric memory hierarchy. Main Idea: Hardware-aware Algorithms. IO-awareness: reducing reads/writes to GPU memory yields significant speedup. FlashAttention: fast and memory-efficient attention algorithm, with no approximation. FlashAttention Adoption Areas. Text Generation.
