ASR: Attention-alike Structural Re-parameterization

TL;DR

This work identifies Stripe Observation, a phenomenon where channel attention values converge to near-constant vectors during training, and leverages it to develop Attention-alike Structural Re-parameterization (ASR). ASR trains with a learnable input $\psi$ feeding the attention module to produce $\mathbf{v}_{\psi,\theta}$, then merges the attention into the backbone at inference so that $\mathbf{B}_{\hat{\theta}} \odot \mathbf{v}_{\psi,\theta}$ becomes a standard SRP transformation, preserving or enhancing performance without extra inference cost. Extensive experiments across backbones (ResNet, ViT, RepVGG, etc.), datasets (ImageNet, STL10, CIFAR), and tasks (classification, segmentation, detection) confirm ASR’s compatibility, robustness, and broad applicability, with ablations clarifying the impact of insertion frequency and learnable input initialization. The work also discusses limitations, notably the method’s current applicability to channel rather than spatial attention, and suggests avenues for future extension to further broaden SRP-assisted deployment of attention mechanisms.

Abstract

The structural re-parameterization (SRP) technique is a novel deep learning technique that achieves interconversion between different network architectures through equivalent parameter transformations. This technique enables the mitigation of the extra costs for performance improvement during training, such as parameter size and inference time, through these transformations during inference, and therefore SRP has great potential for industrial and practical applications. The existing SRP methods have successfully considered many commonly used architectures, such as normalizations, pooling methods, and multi-branch convolution. However, the widely used attention modules which drastically slow inference speed cannot be directly implemented by SRP due to these modules usually act on the backbone network in a multiplicative manner and the modules' output is input-dependent during inference, which limits the application scenarios of SRP. In this paper, we conduct extensive experiments from a statistical perspective and discover an interesting phenomenon Stripe Observation, which reveals that channel attention values quickly approach some constant vectors during training. This observation inspires us to propose a simple-yet-effective attention-alike structural re-parameterization (ASR) that allows us to achieve SRP for a given network while enjoying the effectiveness of the attention mechanism. Extensive experiments conducted on several standard benchmarks demonstrate the effectiveness of ASR in generally improving the performance of existing backbone networks, attention modules, and SRP methods without any elaborated model crafting. We also analyze the limitations and provide experimental and theoretical evidence for the strong robustness of the proposed ASR.

Figures (30)

• Figure 1: The visualization of the Stripe Observation on ImageNet classification. In Section \ref{['sec:settings']}, we shows this observation also holds for segmentation and detection tasks. This phenomenon can be verified on a large number of third-party models with different settings, ensuring its reproducibility and generality. (see Appendix for comprehensive examples) a, after passing through the attention model, the attention values of different images tend to approach a certain value within the same channel, resulting in a "stripe structure". b, Statistical analysis. The standard deviation of attention values for each channel is almost distributed around zero. c, the visualization of the first-order difference (absolute value) for attention value over epoch. Most of the values approach some constants rapidly and consistently.
• Figure 2: The sketch of ASR. Inspired by Stripe Observation, we utilize a learnable vector as input to the attention module for training. In Inference phase, the attention module can be merged into the parameters of the backbone.
• Figure 3: Comparison between attention modules and their ASR in the inference phase on ResNet164 and CIFAR100. The inference speed and #params. of models with ASR are comparable to those of vanilla models, yet have better performance.
• Figure 4: Stripe Observation in other tasks (zoom in for best view).
• Figure 5: The results about the batch noise attack. "BN" denotes batch normalization and "IN" represents instance normalization. Batch noise severely disrupts the results of style transfer. However, even with batch noise present in BN+ASR (SE), it can yield style transfer results comparable to IN without any batch noise, indicating that the proposed ASR can effectively enhance model robustness to mitigate the impact of noise. Zoom in for the best view and see Appendix for more results.