Enhancing Noise Resilience in Face Clustering via Sparse Differential Transformer

TL;DR

The paper tackles noise-induced failures in graph-based face clustering by introducing a prediction-driven Top-$K$ Jaccard similarity to purify neighbor sets, coupled with a Sparse Differential Transformer (SDT) that suppresses noise in attention via Top-$K$ sparsity and differential attention. The approach first adapts neighbor selection before computing Jaccard similarity and then uses SDT (including a Mixture of Experts variant) to robustly estimate central-to-neighbor relationships near the Top-$K$. Extensive experiments on MS-Celeb-1M and cross-domain datasets demonstrate state-of-the-art clustering accuracy and robustness, with notable improvements in $F_P$, $F_B$, and $NMI$, and strong generalization to non-face data. Overall, the combination of adaptive neighbor discovery, enhanced distance metrics, and noise-resilient attention provides a scalable, generalizable framework for robust face clustering and related tasks.

Abstract

The method used to measure relationships between face embeddings plays a crucial role in determining the performance of face clustering. Existing methods employ the Jaccard similarity coefficient instead of the cosine distance to enhance the measurement accuracy. However, these methods introduce too many irrelevant nodes, producing Jaccard coefficients with limited discriminative power and adversely affecting clustering performance. To address this issue, we propose a prediction-driven Top-K Jaccard similarity coefficient that enhances the purity of neighboring nodes, thereby improving the reliability of similarity measurements. Nevertheless, accurately predicting the optimal number of neighbors (Top-K) remains challenging, leading to suboptimal clustering results. To overcome this limitation, we develop a Transformer-based prediction model that examines the relationships between the central node and its neighboring nodes near the Top-K to further enhance the reliability of similarity estimation. However, vanilla Transformer, when applied to predict relationships between nodes, often introduces noise due to their overemphasis on irrelevant feature relationships. To address these challenges, we propose a Sparse Differential Transformer (SDT), instead of the vanilla Transformer, to eliminate noise and enhance the model's anti-noise capabilities. Extensive experiments on multiple datasets, such as MS-Celeb-1M, demonstrate that our approach achieves state-of-the-art (SOTA) performance, outperforming existing methods and providing a more robust solution for face clustering.

Figures (5)

• Figure 1: The facial similarity score with different methods.
• Figure 2: The framework and network architecture of our face clustering method.
• Figure 3: (a) Vanilla Self-attention. (b)Differential Self-attention. (c) Sparse Differential Self-attention (SDT). (d) The Mixture of Experts Sparse Differential Self-attention (MoE-SDT).
• Figure 4: The distance transform function.
• Figure 5: The performance of the Diff Transformer and Vanilla Transformer under different noise ratios.