CARE Transformer: Mobile-Friendly Linear Visual Transformer via Decoupled Dual Interaction

TL;DR

CARE introduces a decoupled, dual-interactive linear attention mechanism to bridge local inductive bias and long-range dependencies in mobile-friendly Vision Transformers. By asymmetrically decoupling features across channels, employing a dynamic memory unit, and using two interaction blocks to fuse information across layers and features, CARE achieves superior accuracy with dramatically reduced GMACs compared to prior linear-attention models. The approach yields strong ImageNet, COCO, and ADE20K results and demonstrates favorable mobile latency, highlighting its practical impact for resource-constrained applications. Limitations include the absence of neural architecture search and evaluation on substantially larger models, which the authors plan to explore in future work.

Abstract

Recently, large efforts have been made to design efficient linear-complexity visual Transformers. However, current linear attention models are generally unsuitable to be deployed in resource-constrained mobile devices, due to suffering from either few efficiency gains or significant accuracy drops. In this paper, we propose a new de\textbf{C}oupled du\textbf{A}l-interactive linea\textbf{R} att\textbf{E}ntion (CARE) mechanism, revealing that features' decoupling and interaction can fully unleash the power of linear attention. We first propose an asymmetrical feature decoupling strategy that asymmetrically decouples the learning process for local inductive bias and long-range dependencies, thereby preserving sufficient local and global information while effectively enhancing the efficiency of models. Then, a dynamic memory unit is employed to maintain critical information along the network pipeline. Moreover, we design a dual interaction module to effectively facilitate interaction between local inductive bias and long-range information as well as among features at different layers. By adopting a decoupled learning way and fully exploiting complementarity across features, our method can achieve both high efficiency and accuracy. Extensive experiments on ImageNet-1K, COCO, and ADE20K datasets demonstrate the effectiveness of our approach, e.g., achieving $78.4/82.1\%$ top-1 accuracy on ImagegNet-1K at the cost of only $0.7/1.9$ GMACs. Codes will be released on \href{https://github.com/zhouyuan888888/CARE-Transformer}{https://github.com/zhouyuan888888/CARE-Transformer}.

Figures (3)

• Figure 1: Visualized comparison of the balance between accuracy, latency, and GMACs for our CARE Transformers and recent typical mobile-friendly models. In the figure, the larger the marker, the more GMACs the model consumes. "SA", "LA", and "GNN" indicate that the methods are based on Self-Attention, Linear Attention, and Graph Neural Networks. For more details, please refer to Table \ref{['tab:imagenet-1k']}.
• Figure 2: Comparisons between (a) the stacked learning approach mlla and (b) our asymmetrical decoupled learning way ($d_1<d_2$) that learns local inductive bias and long-range dependencies separately and further alleviates the quadratic overhead of linear attention to channel dimension, i.e.,$\mathrm{O}(hwd_1^2)$. Local inductive bias is learned by using depth-wise convolutions cpecswinmlla and long-range dependencies are captured by utilizing linear attention mlla.
• Figure 3: Schematic illustration for CARE Transformers, built by our proposed deCoupled duAl-interactive lineaR attEntion (CARE).