SSD: Spatial-Semantic Head Decoupling for Efficient Autoregressive Image Generation

TL;DR

SSD tackles the memory bottleneck in native autoregressive image generation by exploiting a spatial-semantic head dichotomy in visual attention and applying asymmetric KV-cache compression. The method distinguishes spatial-locality heads from semantic-sink heads and uses a sliding window for the former and a top-attended-token retention for the latter, anchored by semantic margin columns. Empirical results show a fivefold memory reduction and a sixfold throughput improvement with negligible quality loss on GenEval and DPG-Bench benchmarks, enabling efficient high-resolution generation on resource-constrained hardware. This work broadens the practicality of unified decoder-only autoregressive architectures for vision by delivering substantial efficiency gains without sacrificing fidelity.

Abstract

Autoregressive image generation models like Janus-Pro produce high-quality images, but at the significant cost of high memory and ever-growing computational demands due to the large number of visual tokens. While KV cache compression has been extensively studied in language modeling, it still remains largely unexplored for the image generation domain. In this work, we begin by identifying a distinct and prominent attention phenomenon, which we term spatial locality and emergent semantic sink. To leverage this key insight, we introduce a novel KV cache compression framework. Specifically, we compress the KV cache for all visual tokens by adaptively decoupling attention heads into two separate types: for spatial-locality heads, our method maintains a short recent token window; for semantic-sink heads, it strategically preserves a compact set of highly-attended tokens. Our extensive experiments demonstrate that the proposed method achieves a 5$\times$ reduction in memory usage and a notable 6.6$\times$ speedup in overall throughput with only minimal visual quality loss, thereby enabling highly efficient native autoregressive image generation on resource-constrained hardware.

Figures (6)

• Figure 1: Visualization of the average attention map and reshaped attention across different generation steps in Janus-Pro-1B over 100 prompts, randomly sampled from Geneval ghosh2023genevalobjectfocusedframeworkevaluating. Our observation reveals: (1) Attention in the vision modality is sparse. (2) There are two types of sparse attention: one termed spatial locality attention, the other termed semantic sink attention. See Appendix \ref{['appendix:more_attn_visualization']} for more details.
• Figure 2: (a) Attention sparsity patterns in Janus-Pro aggregated over 100 instances from Geneval ghosh2023genevalobjectfocusedframeworkevaluating and DPG-Bench hu2024ellaequipdiffusionmodels, revealing different attention sparsity across different heads. (b) Token-wise MSE between conditional and CFG-augmented branches, with higher MSE indicating greater semantic information concentration. The periodic spikes in dense heads (red) correspond to margin column positions, confirming semantic anchoring at spatial boundaries.
• Figure 3: Performance comparison of SSD, Full Cache, StreamingLLM, and H2O on (a,b) GenEval and (c,d) DPG-Bench across varying compression ratios for Janus-Pro-1B and Janus-Pro-7B. SSD achieves performance comparable to Full Cache while significantly reducing memory usage, outperforming both H2O and StreamingLLM on both benchmarks.
• Figure 4: Qualitative visualization of generated images from Janus-Pro-1B and Janus-Pro-7B under varying compression ratios (20% to 50% cache sizes) for SSD and Full Cache. SSD preserves fine details, spatial relationships, and semantic coherence in complex scene generation.
• Figure 5: System performance scaling with batch size. SSD demonstrates superior efficiency compared to full cache, achieving up to 6.6$\times$ throughput improvement and 5$\times$ memory reduction, which highlights its practical advantage in real-world deployment scenarios.