Transforming faces into video stories -- VideoFace2.0
Branko Brkljač, Vladimir Kalušev, Branislav Popović, Milan Sečujski
TL;DR
VideoFace2.0 tackles efficient face re-identification and creation of identity-based video stories in unconstrained TV and multimedia content. It proposes a modular pipeline that combines fast face detection, embedding-based recognition, and passive tracking-by-detection to maintain identity across frames with near real-time performance. Key contributions include showing 18-25 fps on a consumer notebook and a 73-93% reduction in false identities through ablation, along with public code and datasets. The work facilitates rapid generation of multi-modal datasets and structured video outputs for downstream ML tasks, lowering the barrier for production-ready video analysis tools.
Abstract
Face detection and face recognition have been in the focus of vision community since the very beginnings. Inspired by the success of the original Videoface digitizer, a pioneering device that allowed users to capture video signals from any source, we have designed an advanced video analytics tool to efficiently create structured video stories, i.e. identity-based information catalogs. VideoFace2.0 is the name of the developed system for spatial and temporal localization of each unique face in the input video, i.e. face re-identification (ReID), which also allows their cataloging, characterization and creation of structured video outputs for later downstream tasks. Developed near real-time solution is primarily designed to be utilized in application scenarios involving TV production, media analysis, and as an efficient tool for creating large video datasets necessary for training machine learning (ML) models in challenging vision tasks such as lip reading and multimodal speech recognition. Conducted experiments confirm applicability of the proposed face ReID algorithm that is combining the concepts of face detection, face recognition and passive tracking-by-detection in order to achieve robust and efficient face ReID. The system is envisioned as a compact and modular extensions of the existing video production equipment. Presented results are based on test implementation that achieves between 18-25 fps on consumer type notebook. Ablation experiments also confirmed that the proposed algorithm brings relative gain in the reduction of number of false identities in the range of 73%-93%. We hope that the presented work and shared code implementation will stimulate further interest in development of similar, application specific video analysis tools, and lower the entry barrier for production of high-quality multi-modal datasets in the future.