Clustered-patch Element Connection for Few-shot Learning

TL;DR

The paper tackles weak feature representations in few-shot learning caused by semantic mismatch among local patches. It introduces the Clustered-patch Element Connection (CEC) layer, consisting of Patch Cluster (MatMul, Cosine, GCN, Transformer variants) and Element Connection to form a reliable relation map, then aggregates these into the CECNet with CECM, Self-CECM, and CECD distance for robust few-shot classification. Key contributions include three CEC-based modules and a CECE embedding module that extends to few-shot segmentation and detection, achieving state-of-the-art results on standard benchmarks and showing consistent improvements across tasks. The approach provides a principled global-to-local patch linking mechanism that enhances discriminative representations and reliable similarity measurement, with practical impact on fast adaptation in vision tasks like segmentation and detection.

Abstract

Weak feature representation problem has influenced the performance of few-shot classification task for a long time. To alleviate this problem, recent researchers build connections between support and query instances through embedding patch features to generate discriminative representations. However, we observe that there exists semantic mismatches (foreground/ background) among these local patches, because the location and size of the target object are not fixed. What is worse, these mismatches result in unreliable similarity confidences, and complex dense connection exacerbates the problem. According to this, we propose a novel Clustered-patch Element Connection (CEC) layer to correct the mismatch problem. The CEC layer leverages Patch Cluster and Element Connection operations to collect and establish reliable connections with high similarity patch features, respectively. Moreover, we propose a CECNet, including CEC layer based attention module and distance metric. The former is utilized to generate a more discriminative representation benefiting from the global clustered-patch features, and the latter is introduced to reliably measure the similarity between pair-features. Extensive experiments demonstrate that our CECNet outperforms the state-of-the-art methods on classification benchmark. Furthermore, our CEC approach can be extended into few-shot segmentation and detection tasks, which achieves competitive performances.

Figures (4)

• Figure 1: Comparison between traditional Cross Attention and our Clustered-patch Element Connection. The proposed Clustered-patch Element Connection, which utilizes the global info $C^p$ integrated from support feature $P$ to perform element connection with query $Q$ leading to a confident and clear connection, is able to generate a more clear and precise relation map than Cross Attention. The detailed Patch Cluster operation is illustrated in Fig.\ref{['fig:patch']}. The visualization comparisons are referred to Fig.\ref{['fig:visual']}(a).
• Figure 2: Patch Cluster and Element Connection.
• Figure 3: The proposed CECNet framework. The CECM is able to highlight the mutually similar regions, the CECD is utilized to measure similarity of pair-features. And Self-CECM enhances the semantic feature of target object via self-connection.
• Figure 4: (a) The class activation maps on 5-way 1-shot classification, where Embedding belongs to CECNet. (b) The visualizations of our CEC-based relation map $R^Q$.