CASA: Cross-Attention via Self-Attention for Efficient Vision-Language Fusion

TL;DR

The paper introduces CASA, a Cross-Attention via Self-Attention mechanism that injects visual information into LLMs while enabling local text-to-text self-attention in the cross-attention layers. This implicit gating and local attention reduce memory and compute compared to token insertion and standard cross-attention, addressing scalability for high-resolution images and streaming video. They show CASA narrows the gap to insertion-based models across diverse VLM benchmarks, both when training from scratch on text-only LLMs and when adapting existing VLMs, with strong performance on document/chart reading, OCR, and VQA tasks. CASA enables real-time streaming video captioning with low memory growth and latency. They provide extensive experiments, efficiency analyses, and ablations supporting the key role of text-to-text interaction in CASA.

Abstract

Vision-language models (VLMs) are commonly trained by inserting image tokens from a pretrained vision encoder into the textual stream of a language model. This allows text and image information to fully attend to one another within the model, but becomes extremely costly for high-resolution images, long conversations, or streaming videos, both in memory and compute. VLMs leveraging cross-attention are an efficient alternative to token insertion but exhibit a clear performance gap, in particular on tasks involving fine-grained visual details. We find that a key to improving such models is to also enable local text-to-text interaction in the dedicated cross-attention layers. Building on this, we propose CASA, Cross-Attention via Self-Attention, a simple and efficient paradigm which substantially reduces the gap with full token insertion on common image understanding benchmarks, while enjoying the same scalability as cross-attention models when applied to long-context multimodal tasks such as streaming video captioning. For samples and code, please see our project page at https://kyutai.org/casa .

Figures (11)

• Figure 1: Token insertion vs. cross-attention (CA) vs. CASA. In a controlled experiment, we compare various cross-attention (vanilla, + gating, + registers) models with the insertion of visual tokens directly in the language model's text tokens input. We find that while CA methods achieve comparable performance on general visual-question-answering tasks , they significantly lag behind on fine-grained benchmarks such as chart or document understanding. Interestingly, we find this trend to be closely mirrored in the literature when comparing current state-of-the-art cross-attention- and insertion-based models, see \ref{['app:extraspider']}. Based on a detailed analysis, we propose Cross-Attention via Self-Attention (CASA), which largely closes the gap to token insertion.
• Figure 2: CASA (Cross-Attention via Self-Attention) injects visual information through cross-attention layers in which the current text tokens attend to the concatenation of themselves and image tokens, in a causal manner. This provides a natural gating mechanism that outperforms standard cross-attention-based VLM architectures. During training, CASA leverages recent blockwise attention mechanism to remain efficient. At inference, CASA benefits from the same practical advantages as cross-attention and can handle long interleaved image-text sequences without affecting the KV cache and memory usage of the underlying LLM.
• Figure 3: CASA vs. SA. (Left) Instead of inserting image tokens directly into the stream of text tokens at the desired position (dashed lines, bottom), CASA (top) uses block-wise attention to augment groups of tokens with visual information. Compared to token insertion, the cost of self-attention (SA) thus reduces from $\mathcal{O}(T + N)$ to $\mathcal{O}(\max(T_\text{window} + N_\text{window}, N))$, for $T$ text and $N$ image tokens. To illustrate this, we show which tokens a query at a given text position attends to, showing masked tokens at lower opacity. (Right) Attention matrices for CASA and SA layers: while in SA layers text only attends to text, CASA layers apply block-wise attention between text and images.
• Figure 4: Live captioning. We display captions generated by our $\text{CASA}^{ \oplus}_{\text{\scriptsize Qwen2.5-VL}}$-LiveCC model. Each text span is annotated with the corresponding frame's timestamp (top) and the model's delay as [timestamp $|$delay]. As shown in \ref{['fig:liveccmem']}, CASA's outputs are generated in real-time and with little memory building up over time; for further qualitative examples, including insertion-based VLMs, see \ref{['sec:exp-streaming']}.
• Figure 5: Real-time memory usage (reported in MB, log scale) of CASA compared to token insertion techniques with different levels of token compression (via a Q-Former), measured on a single H100 GPU. CASA memory usage builds up more slowly than the token-insertion-based techniques, even with token compression.