AI-Boosted Video Annotation: Assessing the Process Enhancement

TL;DR

The paper investigates enhancing video annotation through AI-powered pre-annotations within a Human-in-the-Loop workflow, aiming to cut labeling effort while maintaining quality. Using Label Studio in a single-iteration scheme and zero-shot AI pre-annotations, the authors tested on the UCF-Crime surveillance dataset with 18 annotators and 30 videos, applying a 6-vote overlap for crowdsourcing quality. Results show an average reduction in annotation time (≈23.11%), with about 70–72% of annotators experiencing faster labeling and improved annotation coherence, supported by CLIP-based semantic clustering and homogenization analyses. The work highlights potential biases introduced by pre-annotations, the importance of annotator learning curves, and broader implications for workforce dynamics and data ownership in AI-augmented annotation pipelines.

Abstract

We explore the enhancement of Human-in-the-Loop video annotation by integrating automatic capabilities to ease the task for annotators and assess their performance. The research delves into the practical implications of the annotation processes, the integration of AI components, and the evaluation of its outcomes. We analyze their impact on efficiency, accuracy, and overall annotation quality. Focusing on the Human-in-the-Loop for video annotation tasks, we implemented a single-iteration scheme using Label Studio and AI-powered zero-shot pre-annotations. Using this framework, we designed a test based on the annotation of the UCF-Crime dataset to discriminate between normal and abnormal activities in video footage. Our results evidence how automatic AI-based pre-annotation can streamline the video annotation workflow, empowering human annotators and optimizing the overall pipeline. Using the pre-annotated data, we observed a 35% reduction in the annotation time for 70% of the annotators with similar quality annotations, compared to the traditional manual annotation task. Results are consistent with asset duration and complexity. We also observed that while annotators rapidly learned to use the tool, the produced annotations are more coherent among annotators and better match the natural clustering of the video frames.

Figures (7)

• Figure 1: Label Studio interface Labelstudio as adapted for the performed experiences and test.
• Figure 2: Diagram of the process followed to evaluate the results of the annotation test and to run the experiences. Videos (to the left) are automatically pre-annotated. These pre-annotations help the annotation process, while input frames are processed using the CLIP image transformer radford_learning_2021. Annotation results (timestamps, labeled classes) and frame embeddings are used to assess annotation results.
• Figure 3: Distribution of annotation times with and without pre-annotations. Bin steps of 20 seconds were employed. The kernel density estimation plot illustrates the trend for both distributions.
• Figure 4: Comparison of annotation times per anonymized user. Each point corresponds to the annotation times of an annotator, and the median of all annotation times with and without pre-annotation has been taken. Figure \ref{['fig:all_ann_a']} contains the total annotation times per video, while in figure \ref{['fig:all_ann_b']}, these times are normalized by the duration of each video. Points above the green area indicate that the median annotation time with pre-annotation is shorter than without, whereas points above the red area suggest the opposite.
• Figure 5: Figure \ref{['fig:time_classe_a']} displays the median annotation time for all videos grouped by the number of non-unique classes. Figure \ref{['fig:time_classe_b']} is the median annotation time per video duration, with intervals set at 5 seconds. In both cases, the data is separated by annotation type.