Scalable Class-Centric Visual Interactive Labeling

TL;DR

This work tackles the scalability and cognitive-load challenges of labeling large datasets with many instances and classes by introducing Class-Centric Visual Interactive Labeling (cVIL). cVIL redefines labeling from assigning classes to instances to selecting instances for a focused class, supported by a visual analytics interface and a four-phase workflow (Bootstrap, Class Selection Guidance, Class-Based Labeling, Residual Labeling). Through a user study and a usage scenario, the authors show that cVIL yields higher final accuracy and improved usability compared to instance-centric approaches, especially as class counts grow. The approach leverages property measures, class-focused guidance, and multi-instance labeling to reduce labeling effort and scale to thousands of classes, while also outlining future work on imbalanced classes and non-visual data modalities.

Abstract

Large unlabeled datasets demand efficient and scalable data labeling solutions, in particular when the number of instances and classes is large. This leads to significant visual scalability challenges and imposes a high cognitive load on the users. Traditional instance-centric labeling methods, where (single) instances are labeled in each iteration struggle to scale effectively in these scenarios. To address these challenges, we introduce cVIL, a Class-Centric Visual Interactive Labeling methodology designed for interactive visual data labeling. By shifting the paradigm from assigning-classes-to-instances to assigning-instances-to-classes, cVIL reduces labeling effort and enhances efficiency for annotators working with large, complex and class-rich datasets. We propose a novel visual analytics labeling interface built on top of the conceptual cVIL workflow, enabling improved scalability over traditional visual labeling. In a user study, we demonstrate that cVIL can improve labeling efficiency and user satisfaction over instance-centric interfaces. The effectiveness of cVIL is further demonstrated through a usage scenario, showcasing its potential to alleviate cognitive load and support experts in managing extensive labeling tasks efficiently.

Figures (14)

• Figure 1: The cVIL prototype contains four main components: The Instance Property View (1) displays the distribution of property measure valuesBernard2021ATO based on all instances predicted for the currently selected focus class. The Instance Similarity View (2) shows the instances of the focus class, along with instances that have the focus class predicted as their top-$k$ class (where $k$ is configurable by the user). This plot also visualizes instances that are already labeled within the projection. To get an overview of label distributions, users can utilize the Class Label View (3). By clicking on a bar, the focus class is changed to that class. The range selection tool below the main bar chart allows users to increase or decrease the number of classes shown or to move the selection entirely. Once a selection is made in the Instance Property View (1) or Instance Similarity View (2), the selected instances are displayed in the Instance Labeling View (4), where users can label either the entire selection or individual instances.
• Figure 2: t-SNE projection of high dimensional embeddings, used for the iVIL approach in our user study. The visualizations are based on embeddings generated by a pre-trained DINO Caron2021DINO model, derived from subsets of the CelebA dataset defined by distinct attributes: people's hair color and whether or not they wear glasses. To ensure clarity, we filtered out images classified as having multiple hair colors, only considering those exclusively assignable to one hair color category. The first figure showcases a representative sample of the resulting dataset, featuring 1,000 images for each of the four distinct hair color classes. The second figure shows the projections of the embeddings from people with and without glasses.
• Figure 3: The conceptual cVIL workflow consists of four phases. The process is bounded by a Bootstrap phase and a Residual Labeling phase, both with a traditional instance-centricc focus. In contrast, the core part of the workflow introduces a class-centric focus. In two highly iterative phases, users find means to identify most promising classes to be labeled next, possibly with computational support (Class Selection Guidance), and conduct the Class-Based Labeling.
• Figure 4: The Class Label View uses a stacked bar chart to assess the distribution of instances per class and label types, color-coded as Unlabeled (blue), Batch Labeled (orange), and Manually Labeled (green). Below, a minimized version of the bar chart shows the distribution for all classes (scalable for many dozens), with the current subset highlighted with gray area. In the example, the Butterfly class stands out with high instance count, while other classes have comparatively smaller sizes. The within-class assessments reveal differing proportions of label types; for example, the Octopus class has very few batch-labeled instances -- something the user may want to address next.
• Figure 5: The Instance Property View shows a kernel density estimation of the property measure output to visualize the distribution of values. We chose the valance of property measures such that instances with small values are more likely to be prototypical instances of the class and instances with larger values are likely to be more atypical, allowing users to quickly gauge the system's performance. The x-axis is log-transformed, allowing the visualization to show a large range of values. In this instance, the Eccentricity score is visualized, which is very skewed, which indicates that most remaining unlabeled samples for this class are distributed unevenly around the already labeled points.