VideoClusterNet: Self-Supervised and Adaptive Face Clustering For Videos

TL;DR

A novel video face clustering approach that learns to adapt a generic face ID model to new video face tracks in a fully self-supervised fashion and a parameter-free clustering algorithm that is capable of automatically adapting to the finetuned model's embedding space for any input video is proposed.

Abstract

With the rise of digital media content production, the need for analyzing movies and TV series episodes to locate the main cast of characters precisely is gaining importance.Specifically, Video Face Clustering aims to group together detected video face tracks with common facial identities. This problem is very challenging due to the large range of pose, expression, appearance, and lighting variations of a given face across video frames. Generic pre-trained Face Identification (ID) models fail to adapt well to the video production domain, given its high dynamic range content and also unique cinematic style. Furthermore, traditional clustering algorithms depend on hyperparameters requiring individual tuning across datasets. In this paper, we present a novel video face clustering approach that learns to adapt a generic face ID model to new video face tracks in a fully self-supervised fashion. We also propose a parameter-free clustering algorithm that is capable of automatically adapting to the finetuned model's embedding space for any input video. Due to the lack of comprehensive movie face clustering benchmarks, we also present a first-of-kind movie dataset: MovieFaceCluster. Our dataset is handpicked by film industry professionals and contains extremely challenging face ID scenarios. Experiments show our method's effectiveness in handling difficult mainstream movie scenes on our benchmark dataset and state-of-the-art performance on traditional TV series datasets.

Figures (6)

• Figure 1: Select hard case clusters predicted using our algorithm from within MovieFaceCluster dataset. Trivial face represents an easy ID sample for each cluster. The term “varying parameter” depicts the dominant image attributes that are particularly challenging for a given face crop. It is not part of the dataset annotations but is depicted for enhanced reader understanding.
• Figure 2: Overview of VideoClusterNet: Stage 1: Given the temporal continuity in the video domain, faces detected in consecutive frames are first locally grouped into tracks using a motion tracking algorithm. Stage 2: A large-scale pre-trained face ID model is finetuned on these tracks using temporal self-supervision (w/ only positive pairing), via learning through natural and augmented face variations within each track. The finetuning is bootstrapped by soft-matching tracks across common identities. Performing these two steps alternatively helps the model better understand the given set of faces. Stage 3: An agglomerative clustering algorithm based on a model-learned similarity metric groups common identity tracks.
• Figure 3: Self-Supervised Model Finetuning: Face crop pair sampled from within same/matched track(s) is passed through a student and teacher branch, respectively. Gradients w.r.t. similarity loss are backpropagated only through the student branch, while the teacher weights are updated as moving average of student weights. Random augmentation set includes horizontal flipping, rotation, and color temperature variations.
• Figure 4: Coarse Face Track Matching: A Multivariate Gaussian is fitted to every track crop distribution. Then, a custom track matching threshold is computed using outlier crops pdf values. Neighboring tracks having a mean pdf value higher than a custom threshold are soft-matched with the given track.
• Figure 5: Comparative t-SNE embedding visualizations laurens2008_tsne on MovieFaceCluster:The Hidden Soldier dataset. Left: Ground Truth (GT), Right: Our method. Each dot in the diagram above represents the finetuned model's extracted embedding for a given face crop $I_{t_{n}}$ in a given track's sampled crop set $t$. Face embeddings assigned to a given color constitute a single cluster. Our method predicts almost perfectly the cluster designations (22 clusters) w.r.t. ground truth (21 clusters). Also note that our method correctly assigns cluster IDs to certain outlier tracks in GT. Such tracks pose a significant face clustering challenge owing to their distance from respective GT cluster centers.