Leveraging Human-Machine Interactions for Computer Vision Dataset Quality Enhancement

TL;DR

This work tackles the problem of dataset quality in single-label computer vision benchmarks by revealing substantial multi-label content in ImageNetV2 and its impact on model evaluation. It introduces Multilabelfy, a four-stage human-in-the-loop framework that combines model-driven label proposals with targeted human annotation, disagreement analysis, and refinement, culminating in a richer, multi-label reassessment of ImageNetV2. Key findings include $47.88\%$ of ImageNetV2 images having multiple valid labels and a positive, near-linear relationship between Top-1 and ReaL accuracy across $57$ models ($R^2 = 75.69\%$, slope $0.5788$), underscoring the importance of multi-label truth in benchmarking. The authors provide an open-source implementation and an accessible workflow to help smaller labs improve dataset integrity, enabling more robust model evaluation and generalization insights in vision tasks.

Abstract

Large-scale datasets for single-label multi-class classification, such as \emph{ImageNet-1k}, have been instrumental in advancing deep learning and computer vision. However, a critical and often understudied aspect is the comprehensive quality assessment of these datasets, especially regarding potential multi-label annotation errors. In this paper, we introduce a lightweight, user-friendly, and scalable framework that synergizes human and machine intelligence for efficient dataset validation and quality enhancement. We term this novel framework \emph{Multilabelfy}. Central to Multilabelfy is an adaptable web-based platform that systematically guides annotators through the re-evaluation process, effectively leveraging human-machine interactions to enhance dataset quality. By using Multilabelfy on the ImageNetV2 dataset, we found that approximately $47.88\%$ of the images contained at least two labels, underscoring the need for more rigorous assessments of such influential datasets. Furthermore, our analysis showed a negative correlation between the number of potential labels per image and model top-1 accuracy, illuminating a crucial factor in model evaluation and selection. Our open-source framework, Multilabelfy, offers a convenient, lightweight solution for dataset enhancement, emphasizing multi-label proportions. This study tackles major challenges in dataset integrity and provides key insights into model performance evaluation. Moreover, it underscores the advantages of integrating human expertise with machine capabilities to produce more robust models and trustworthy data development. The source code for Multilabelfy will be available at https://github.com/esla/Multilabelfy. \keywords{Computer Vision \and Dataset Quality Enhancement \and Dataset Validation \and Human-Computer Interaction \and Multi-label Annotation.}

Figures (7)

• Figure 1: Overview of the proposed framework for enhancing computer vision datasets from single-label to multi-label, enabling a more comprehensive capture of their descriptions.
• Figure 2: The user interface of the annotation platform. It showcases key features like label presentation in groups of five, a single checkbox per proposed label, scrollable sample images, and click-to-enlarge functionality for detailed inspection of images. These features are designed to streamline the annotation process and efficiently accommodate multi-label data annotation.
• Figure 3: The distribution of images based on the number of labels assigned to them during our annotation process.
• Figure 4: Scatterplot of ReaL accuracy versus top-1 accuracy for $57$ top-performing DNN models, pre-trained either exclusively on the ImageNet-1k dataset or additionally on external datasets.
• Figure 5: Heatmaps displaying top-1 accuracy (top) for five randomly selected models evaluated on our multi-labeled ImageNetV2 dataset, and the half-width of the $95\%$ confidence interval (bottom) associated with these accuracies. Red cells without numbers represent NaN values due to sets with one or no images for a given label count.