In-Context Learning in Linear vs. Quadratic Attention Models: An Empirical Study on Regression Tasks

TL;DR

This paper empirically study how the similarities and limitations of linear attention relative to quadratic attention in their ICL behavior on the canonical linear-regression task of Garg et al.

Abstract

Recent work has demonstrated that transformers and linear attention models can perform in-context learning (ICL) on simple function classes, such as linear regression. In this paper, we empirically study how these two attention mechanisms differ in their ICL behavior on the canonical linear-regression task of Garg et al. We evaluate learning quality (MSE), convergence, and generalization behavior of each architecture. We also analyze how increasing model depth affects ICL performance. Our results illustrate both the similarities and limitations of linear attention relative to quadratic attention in this setting.

Figures (3)

• Figure 1: Training and Testing loss. Top row shows shallower networks (1 and 3 layers), while the bottom row shows the deeper 6-layer model.
• Figure 2: Training and Testing loss for the Linear Attention Transformer
• Figure 3: Isotropic vs. anisotropic performance across depths.