Efficient Modulation for Vision Networks

TL;DR

EfficientMod addresses the efficiency limitations of Vision Transformers by introducing a unified convolutional modulation block that fuses large-receptive-field context with feature projection through simple element-wise multiplication. The method grounds itself in a streamlined modulation mechanism, inspired by VAN and FocalNet, and refines it into a slim, fast block that scales well on hardware. By constructing pure-convolutional EfficientMod backbones and a hybrid variant with self-attention in later stages, the approach achieves strong ImageNet-1K performance and superior latency-accuracy trade-offs, while delivering notable gains on semantic segmentation (ADE20K) and object detection/instance segmentation (COCO). The work demonstrates practical design principles for efficient vision networks, including reduced fragmentation, prioritized late-stage capacity, and compatibility with attention-based modules for further gains. Overall, EfficientMod offers a robust, scalable path toward deployable, high-accuracy vision models on edge devices and real-time systems.

Abstract

In this work, we present efficient modulation, a novel design for efficient vision networks. We revisit the modulation mechanism, which operates input through convolutional context modeling and feature projection layers, and fuses features via element-wise multiplication and an MLP block. We demonstrate that the modulation mechanism is particularly well suited for efficient networks and further tailor the modulation design by proposing the efficient modulation (EfficientMod) block, which is considered the essential building block for our networks. Benefiting from the prominent representational ability of modulation mechanism and the proposed efficient design, our network can accomplish better trade-offs between accuracy and efficiency and set new state-of-the-art performance in the zoo of efficient networks. When integrating EfficientMod with the vanilla self-attention block, we obtain the hybrid architecture which further improves the performance without loss of efficiency. We carry out comprehensive experiments to verify EfficientMod's performance. With fewer parameters, our EfficientMod-s performs 0.6 top-1 accuracy better than EfficientFormerV2-s2 and is 25% faster on GPU, and 2.9 better than MobileViTv2-1.0 at the same GPU latency. Additionally, our method presents a notable improvement in downstream tasks, outperforming EfficientFormerV2-s by 3.6 mIoU on the ADE20K benchmark. Code and checkpoints are available at https://github.com/ma-xu/EfficientMod.

Figures (10)

• Figure 1: Comparison of Transformer, abstracted modulation design, and our EfficientMod block. $\odot$ is element-wise multiplication and $\otimes$ means matrix multiplication. Compared to Transformer and abstracted modulation, our unified block efficiently modulates the projected values ($V$) via a simple context modeling design ($\texttt{CTX}$). Dimension number is indicated in (c) to aid comprehension.
• Figure 2: From Repeat to Reshape, EfficientMod-s GPU latency decreases 5.1% and CPU latency increases 21.8%.
• Figure 3: The trade-off between ONNX GPU latency and accuracy.
• Figure 4: We directly visualize the forward context modeling results as shown in Eq. \ref{['eq:our_ctx_branch']}. The visualization results suggest that our context modeling can emphasize the conspicuous context. No backward gradient is required as in Class Activation Map zhou2016learning.
• Figure 5: Ablation studies based on EfficientMod-s-Conv w/o attention.