CLUENet: Cluster Attention Makes Neural Networks Have Eyes

TL;DR

CLUENet introduces a transparent clustering-based vision architecture that overcomes fixed receptive-field limitations by using Global Feature Clustering with a Temperature-Scaled Cosine Attention and a gated residuals mechanism. It couples global-soft aggregation with hard, shared assignment and an improved cluster pooling strategy to maintain training stability and mitigate gradient vanishing. The approach yields strong classification performance on CIFAR-100 and Mini-ImageNet while offering intrinsic interpretability through cluster visualizations and receptive-field analysis. This work demonstrates that flexible, cluster-based representations can rival mainstream models in accuracy while enhancing transparency and efficiency for visual semantic understanding.

Abstract

Despite the success of convolution- and attention-based models in vision tasks, their rigid receptive fields and complex architectures limit their ability to model irregular spatial patterns and hinder interpretability, therefore posing challenges for tasks requiring high model transparency. Clustering paradigms offer promising interpretability and flexible semantic modeling, but suffer from limited accuracy, low efficiency, and gradient vanishing during training. To address these issues, we propose CLUster attEntion Network (CLUENet), an transparent deep architecture for visual semantic understanding. We propose three key innovations include (i) a Global Soft Aggregation and Hard Assignment with a Temperature-Scaled Cosin Attention and gated residual connections for enhanced local modeling, (ii) inter-block Hard and Shared Feature Dispatching, and (iii) an improved cluster pooling strategy. These enhancements significantly improve both classification performance and visual interpretability. Experiments on CIFAR-100 and Mini-ImageNet demonstrate that CLUENet outperforms existing clustering methods and mainstream visual models, offering a compelling balance of accuracy, efficiency, and transparency.

Figures (6)

• Figure 1: Different paradigms of visual models. Top: convolutional paradigms using convolution for local information modeling. Middle: attention paradigms leveraging positional embeddings and attention for global feature modeling. Bottom: clustering paradigms employing coordinate-guided clustering for semantic modeling. (Representative models are shown on the right, w/ 5M to 15M parameters.)
• Figure 2: (a) Overall architecture of the CLUENet with a four-stage pyramid network; (b) The key components within the Global Feature Clustering (GFC) block, illustrating Global and Soft Feature Aggregation (GSFA) that updates cluster centers from all pixels, and Hard and Shared Feature Dispatching (HSFD) that updates pixel features according to their assigned cluster centers; (c) The Improved Cluster Pooling (ICP) block, depicting how pixel features are grouped into clusters in similarity space while preserving hierarchical structure.
• Figure 3: The details of the Global Feature Clustering (GFC) block. Global and Soft Feature Aggregation (GSFA) includes center initialization, Temperature-Scaled Cosine Attention, and Gated Fusion Mechanism. Hard and Shared Feature Dispatching (HSFD) includes multi-head query projection, hard clustering, and assignment shared across blocks.
• Figure 4: Cluster pooling configurations. (a) In FEC, $\operatorname{proj_f}$ only guides pixel selection and does not update, (b) connecting $\operatorname{proj_f}$ and $\operatorname{proj_v}$ avoids gradient issues but offers limited performance gains, (c) The proposed cluster pooling adopts a two-layer perceptron for $\operatorname{proj_v}$, enabling effective training and improved performance.
• Figure 5: Visualization of (a) the clustering results of semantic heads at each of the four stages, along with the global receptive field map w.r.t. the final classification decision, and (b) the global receptive field map w.r.t. different cluster numbers.