Efficient Attention using a Fixed-Size Memory Representation

TL;DR

The paper tackles the high computational cost of standard content-based attention in seq2seq models by introducing a fixed-size memory representation that stores K attention contexts during encoding. The decoder then uses a cheap linear combination over this memory, yielding a total complexity of $O(KD(|S|+|T|))$, which provides inference speedups, especially for longer sequences, while maintaining on-par accuracy with traditional attention on translation tasks. Across toy and real-world MT experiments, the method achieves competitive BLEU scores and demonstrates faster decoding, with position encodings further improving performance and encouraging diverse, interpretable alignments. Visualization confirms that the memory learns distinct, meaningful source-region alignments, validating the approach as a practical, scalable alternative for long-sequence tasks like MT and beyond.

Abstract

The standard content-based attention mechanism typically used in sequence-to-sequence models is computationally expensive as it requires the comparison of large encoder and decoder states at each time step. In this work, we propose an alternative attention mechanism based on a fixed size memory representation that is more efficient. Our technique predicts a compact set of K attention contexts during encoding and lets the decoder compute an efficient lookup that does not need to consult the memory. We show that our approach performs on-par with the standard attention mechanism while yielding inference speedups of 20% for real-world translation tasks and more for tasks with longer sequences. By visualizing attention scores we demonstrate that our models learn distinct, meaningful alignments.

Figures (8)

• Figure 1: Memory Attention model architecture. $K$ attention vectors are predicted during encoding, and a linear combination is chosen during decoding. In our example, $K=3$.
• Figure 2: Surface for the position encodings.
• Figure 3: Training Curves for the Toy Copy task
• Figure 4: Comparing training curves for en-fi and en-tr with sigmoid encoder scoring and softmax decoder scoring and position encoding. Note that en-tr curves converged very quickly.
• Figure 5: Comparing training curves for en-fi for different encoder/decoder scoring functions for our models at $K=64$.