Learning 1D Causal Visual Representation with De-focus Attention Networks

TL;DR

This paper proposes De-focus Attention Networks, which employ learnable bandpass filters to create varied attention patterns and demonstrates that 1D causal visual representation can perform comparably to 2D non-causal representation in tasks such as global perception, dense prediction, and multi-modal understanding.

Abstract

Modality differences have led to the development of heterogeneous architectures for vision and language models. While images typically require 2D non-causal modeling, texts utilize 1D causal modeling. This distinction poses significant challenges in constructing unified multi-modal models. This paper explores the feasibility of representing images using 1D causal modeling. We identify an "over-focus" issue in existing 1D causal vision models, where attention overly concentrates on a small proportion of visual tokens. The issue of "over-focus" hinders the model's ability to extract diverse visual features and to receive effective gradients for optimization. To address this, we propose De-focus Attention Networks, which employ learnable bandpass filters to create varied attention patterns. During training, large and scheduled drop path rates, and an auxiliary loss on globally pooled features for global understanding tasks are introduced. These two strategies encourage the model to attend to a broader range of tokens and enhance network optimization. Extensive experiments validate the efficacy of our approach, demonstrating that 1D causal visual representation can perform comparably to 2D non-causal representation in tasks such as global perception, dense prediction, and multi-modal understanding. Code is released at https://github.com/OpenGVLab/De-focus-Attention-Networks.

Figures (7)

• Figure 1: Visualizations of (a) Attention Map and (b) Gradient Map of different models, including Non-causal ViT, Causal ViT, Causal Mamba and our De-focus Attention Network (Mamba-based). The results are from the 11th layer of ViT (12 in total) and 22nd layer of Mamba (24 in total). (a) The approximated attention maps of all image tokens: The row and column axes represent the query and key token index respectively. Brighter color indicates larger attention values. (b) The gradient maps of each image token input after back-propagation: Redder colors indicate larger gradient norms. See Appendix \ref{['append:vis']} for more visualizations on different layers.
• Figure 2: Architecture of our De-focus Attention Network.Left: Detailed architecture of De-focus Attention Block: The input tokens are projected to $Q, K$, and other parameters required by certain causal attention layer (e.g. Transformer or Mamba). $\Delta$ is data-dependent in De-focus Mamba, while is set to 1 in De-focus ViT. Learnable decay and learnable relative position embeddings form a learnable bandpass filter and are calculated before being fed into the causal attention layer. Parameter $\lambda$ in De-focus ViT corresponds to $A$ in this figure. Right: Overall architecture of De-focus Attention Network: Drop paths are incorporated after each De-focus Attention Block. All output image tokens are passed through Average Pooling and a fully connected layer to produce the auxiliary loss.
• Figure 3: Qualitative ablation results of using scheduled drop path and auxiliary loss.(a)-(b): The receptive fields of our model trained with and without scheduled drop path. The scheduled drop path strategy enables a larger receptive field, facilitating the capture of denser semantic details. (c)-(d): The backward gradient maps of our model trained with and without auxiliary loss. When trained with the auxiliary loss, the model can attend to denser and more diverse image tokens, particularly those at the front of the sequence.
• Figure 4: Visualizations of (a) Attention Map and (b) Gradient Map of different models, including non-causal ViT, causal ViT, Causal Mamba and our De-focus Attention Network (Mamba-based). The results are from the 3rd layer of ViT (12 in total) and 6th layer of Mamba (24 in total). (a) The approximated attention maps of all image tokens: The row and column axis represent the query and key token index respectively. Brighter color indicates larger attention values. (b) The gradient maps of each image token input after back-propagation: Redder colors indicate larger gradient norms.
• Figure 5: Visualizations of (a) Attention Map and (b) Gradient Map of different models, including non-causal ViT, causal ViT, Causal Mamba and our De-focus Attention Network (Mamba-based). The results are from the 6th layer of ViT (12 in total) and 12th layer of Mamba (24 in total). (a) The approximated attention maps of all image tokens: The row and column axis represent the query and key token index respectively. Brighter color indicates larger attention values. (b) The gradient maps of each image token input after back-propagation: Redder colors indicate larger gradient norms.