Lightweight Channel Attention for Efficient CNNs

TL;DR

This paper systematically compares Squeeze-and-Excitation (SE), Efficient Channel Attention (ECA), and a new Lite Channel Attention (LCA) across ResNet-18 and MobileNetV2 on CIFAR-10 to understand efficiency-accuracy trade-offs in resource-constrained settings. LCA builds on ECA by introducing grouped 1D convolutions with adaptive kernel sizing to reduce parameters while preserving effective channel recalibration. Empirical results show SE offers the highest accuracy on ResNet-18 but with higher latency, while ECA and LCA achieve competitive accuracy with minimal parameter overhead and favorable latency, especially for mobile architectures. The study provides an open, hardware-aware benchmarking framework and highlights that localized channel interactions can match full cross-channel attention in many practical scenarios, guiding deployment decisions in constrained environments.

Abstract

Attention mechanisms have become integral to modern convolutional neural networks (CNNs), delivering notable performance improvements with minimal computational overhead. However, the efficiency accuracy trade off of different channel attention designs remains underexplored. This work presents an empirical study comparing Squeeze and Excitation (SE), Efficient Channel Attention (ECA), and a proposed Lite Channel Attention (LCA) module across ResNet 18 and MobileNetV2 architectures on CIFAR 10. LCA employs adaptive one dimensional convolutions with grouped operations to reduce parameter usage while preserving effective attention behavior. Experimental results show that LCA achieves competitive accuracy, reaching 94.68 percent on ResNet 18 and 93.10 percent on MobileNetV2, while matching ECA in parameter efficiency and maintaining favorable inference latency. Comprehensive benchmarks including FLOPs, parameter counts, and GPU latency measurements are provided, offering practical insights for deploying attention enhanced CNNs in resource constrained environments.

Figures (5)

• Figure 1: Inference latency comparison by model and attention mechanism.
• Figure 2: Accuracy-latency trade-off across models and attention mechanisms.
• Figure 3: Accuracy-parameter trade-off showing efficiency of lightweight attention.
• Figure 4: FLOPs-accuracy trade-off demonstrating computational efficiency.
• Figure 5: Throughput-accuracy trade-off showing practical deployment metrics.