Video Annotator: A framework for efficiently building video classifiers using vision-language models and active learning

TL;DR

Video Annotator (VA) addresses the high cost and inefficiency of traditional video annotation by integrating active learning with vision-language model backbones and direct domain-expert involvement in a continuous annotation-deployment loop. The framework uses pre-trained video-text embeddings, text-to-video search for bootstrap labeling, and lightweight binary classifiers per label within a four-feed active-learning interface to rapidly refine models. VA introduces Model quality and Data diversity scores, and demonstrates substantial improvements in $AP$ (up to about $8.3$ points) over competitive baselines across 56 tasks, supported by a dataset of 153k annotations and accompanying code. This approach enables fast, trusted, scalable video classification and suggests broad applicability to other media modalities, with potential for practical impact in search, personalization, and content understanding.

Abstract

High-quality and consistent annotations are fundamental to the successful development of robust machine learning models. Traditional data annotation methods are resource-intensive and inefficient, often leading to a reliance on third-party annotators who are not the domain experts. Hard samples, which are usually the most informative for model training, tend to be difficult to label accurately and consistently without business context. These can arise unpredictably during the annotation process, requiring a variable number of iterations and rounds of feedback, leading to unforeseen expenses and time commitments to guarantee quality. We posit that more direct involvement of domain experts, using a human-in-the-loop system, can resolve many of these practical challenges. We propose a novel framework we call Video Annotator (VA) for annotating, managing, and iterating on video classification datasets. Our approach offers a new paradigm for an end-user-centered model development process, enhancing the efficiency, usability, and effectiveness of video classifiers. Uniquely, VA allows for a continuous annotation process, seamlessly integrating data collection and model training. We leverage the zero-shot capabilities of vision-language foundation models combined with active learning techniques, and demonstrate that VA enables the efficient creation of high-quality models. VA achieves a median 6.8 point improvement in Average Precision relative to the most competitive baseline across a wide-ranging assortment of tasks. We release a dataset with 153k labels across 56 video understanding tasks annotated by three professional video editors using VA, and also release code to replicate our experiments at: http://github.com/netflix/videoannotator.

Figures (11)

• Figure 1: Functional view of a binary video classifier. A few-second clip from "Operation Varsity Blues: The College Admissions Scandal" operation_varsity_blues is passed to a binary classifier for detecting the "establishing shots" mittal2007detecting label. The classifier outputs a very high score, indicating that the video clip is very likely an establishing shot.
• Figure 2: Three video clips and the corresponding binary classifier scores for three video understanding labels. Note that these labels are not mutually exclusive. Video clips are from Operation Varsity Blues: The College Admissions Scandal operation_varsity_blues, 6 Underground 6_underground, and Leave The World Behind leave_the_world_behind, respectively.
• Figure 3: The main annotation process of Video Annotator (VA) involves three steps, illustrated by the green components. In step 1, the annotator retrieves video clips from text queries which are encoded using $e_t$. In Step 2, the clips are labeled and used to build a classifier for further refinement. Finally, step 3 is an opportunity to review all annotated clips. Note that this process is rarely linear, and the annotator can easily navigate between steps.
• Figure 4: Step 1: Text-to-video search to bootstrap the annotation process.
• Figure 5: Step 2 - Active Learning (AL) loop. The annotator clicks on build, which initiates classifier training and scoring of all clips in $C$. Scored clips are displayed in four feeds.