Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Visual Autoregressive Transformers Must Use $Ω(n^2 d)$ Memory

Yang Cao, Xiaoyu Li, Yekun Ke, Yingyu Liang, Zhenmei Shi, Zhao Song

TL;DR

The paper tackles the memory bottleneck in Visual Autoregressive Transformers by formalizing the KV-cache compression problem and proving a fundamental $\Omega(n^2 d)$ memory lower bound for sequential visual token generation when $d = \Omega(\log n)$. Its core method combines reductions from one-way communication problems (Index and Multi-Index) with Johnson-Lindenstrauss projections to show that subquadratic memory usage is unattainable without additional structural constraints. The results hold for both exact and $(1\pm\eta)$-approximate attention, highlighting a theoretical limit on memory efficiency for VAR KV-cache. The work also discusses sparsity priors as potential mitigations, clarifying when practical reductions may be possible and guiding future design of memory-efficient VAR architectures.

Abstract

A fundamental challenge in Visual Autoregressive models is the substantial memory overhead required during inference to store previously generated representations. Despite various attempts to mitigate this issue through compression techniques, prior works have not explicitly formalized the problem of KV-cache compression in this context. In this work, we take the first step in formally defining the KV-cache compression problem for Visual Autoregressive transformers. We then establish a fundamental negative result, proving that any mechanism for sequential visual token generation under attention-based architectures must use at least $Ω(n^2 d)$ memory, when $d = Ω(\log n)$, where $n$ is the number of tokens generated and $d$ is the embedding dimensionality. This result demonstrates that achieving truly sub-quadratic memory usage is impossible without additional structural constraints. Our proof is constructed via a reduction from a computational lower bound problem, leveraging randomized embedding techniques inspired by dimensionality reduction principles. Finally, we discuss how sparsity priors on visual representations can influence memory efficiency, presenting both impossibility results and potential directions for mitigating memory overhead.

Visual Autoregressive Transformers Must Use $Ω(n^2 d)$ Memory

TL;DR

The paper tackles the memory bottleneck in Visual Autoregressive Transformers by formalizing the KV-cache compression problem and proving a fundamental memory lower bound for sequential visual token generation when . Its core method combines reductions from one-way communication problems (Index and Multi-Index) with Johnson-Lindenstrauss projections to show that subquadratic memory usage is unattainable without additional structural constraints. The results hold for both exact and -approximate attention, highlighting a theoretical limit on memory efficiency for VAR KV-cache. The work also discusses sparsity priors as potential mitigations, clarifying when practical reductions may be possible and guiding future design of memory-efficient VAR architectures.

Abstract

A fundamental challenge in Visual Autoregressive models is the substantial memory overhead required during inference to store previously generated representations. Despite various attempts to mitigate this issue through compression techniques, prior works have not explicitly formalized the problem of KV-cache compression in this context. In this work, we take the first step in formally defining the KV-cache compression problem for Visual Autoregressive transformers. We then establish a fundamental negative result, proving that any mechanism for sequential visual token generation under attention-based architectures must use at least memory, when , where is the number of tokens generated and is the embedding dimensionality. This result demonstrates that achieving truly sub-quadratic memory usage is impossible without additional structural constraints. Our proof is constructed via a reduction from a computational lower bound problem, leveraging randomized embedding techniques inspired by dimensionality reduction principles. Finally, we discuss how sparsity priors on visual representations can influence memory efficiency, presenting both impossibility results and potential directions for mitigating memory overhead.

Paper Structure

This paper contains 22 sections, 7 theorems, 20 equations, 2 figures.

Table of Contents

  1. Introduction
  2. Our Contributions
  3. Roadmap
  4. Related Work
  5. Visual Autoregressive Models
  6. KV Cache Compression
  7. Theoretical Limitations of Transformer.
  8. Preliminary
  9. KV-Cache Compression Problem
  10. Visual Autoregressive Transformer
  11. KV-Cache Compression in VAR
  12. Johnson-Lindenstrauss Projection
  13. Sub-quadratic Space for KV Cache Compression is Impossible
  14. Index Problem
  15. Hardness of Index Problem
  16. ...and 7 more sections

Key Result

Theorem 1.4

Let $N$ denote the total number of feature map scales generated by $\mathop{\mathrm{\mathsf{VAR}}}\nolimits$ transformer, where the height and width at the $N$-th level satisfy $h_N = w_N = n$. There exists some universal constant $C_u > 1$ and for $d \geq C_u \log n$, any algorithm that can, with p must use at least $\Omega(n^2 d)$ bits of memory.

Figures (2)

  • Figure 1: Example of the Pyramid Up-Interpolation Layer $\Phi_{{\rm up},2}$ used in the model.
  • Figure 2: KV Cache Schematic of the $\mathop{\mathrm{\mathsf{VAR}}}\nolimits$ Transformer. We present the example of the 3rd iteration of the VAR Transformer: Before the iteration, the KV cache contains $K_1, V_1$ vectors from the initial input and $K_2, V_2$ vectors from tokens generated via up-interpolation from Scale $Z_1$. During this iteration, the model computes new keys, queries, and values for tokens derived via up-interpolation from Scale $Z_2$, and appends the updated keys/values to the cache for autoregressive generation.

Theorems & Definitions (22)

  • Definition 1.2: KV Cache Compress Problem in $\mathop{\mathrm{\mathsf{VAR}}}\nolimits$ Transformer, informal version of Definition \ref{['def:kv-cache_var:formal']}
  • Theorem 1.4: Space Complexity Lower Bounds for Key-Value Cache in Precise Attention Computation, informal version of Theorem \ref{['thm:space_lower_bound_precise:formal']}
  • Definition 3.1: KV Cache Compression Problem
  • Remark 3.2
  • Definition 3.3: Up-interpolation Layer
  • Definition 3.4: Pyramid Up-Interpolation Layer $\Phi_{{\rm}}$
  • Definition 3.5: Single Attention Layer
  • Definition 3.6: $\mathop{\mathrm{\mathsf{VAR}}}\nolimits$ Transformer
  • Definition 3.7: KV Cache Compress Problem in the $\mathop{\mathrm{\mathsf{VAR}}}\nolimits$ Transformer
  • Definition 3.8: Definition 3 in w14_book, JL-tranform
  • ...and 12 more