Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Can Vision-Language Models Replace Human Annotators: A Case Study with CelebA Dataset

Haoming Lu, Feifei Zhong

TL;DR

This study assesses whether Vision-Language Models can replace human annotators for image attribute labeling, using the CelebA dataset and the LLaVA-NeXT model. It evaluates AI-generated annotations on 1000 CelebA images across 40 binary attributes, incorporating a re-annotation step and majority voting to measure quality against human labels. Results show AI-human agreement of 79.5%, rising to 89.1% with re-annotations, with higher concordance on objective attributes, and AI labeling costing under 1% of manual annotation. The findings demonstrate potential for cost-effective AI annotations in specific tasks, while highlighting the need for scaling to more complex labels and addressing model biases.

Abstract

This study evaluates the capability of Vision-Language Models (VLMs) in image data annotation by comparing their performance on the CelebA dataset in terms of quality and cost-effectiveness against manual annotation. Annotations from the state-of-the-art LLaVA-NeXT model on 1000 CelebA images are in 79.5% agreement with the original human annotations. Incorporating re-annotations of disagreed cases into a majority vote boosts AI annotation consistency to 89.1% and even higher for more objective labels. Cost assessments demonstrate that AI annotation significantly reduces expenditures compared to traditional manual methods -- representing less than 1% of the costs for manual annotation in the CelebA dataset. These findings support the potential of VLMs as a viable, cost-effective alternative for specific annotation tasks, reducing both financial burden and ethical concerns associated with large-scale manual data annotation. The AI annotations and re-annotations utilized in this study are available on https://github.com/evev2024/EVEV2024_CelebA.

Can Vision-Language Models Replace Human Annotators: A Case Study with CelebA Dataset

TL;DR

This study assesses whether Vision-Language Models can replace human annotators for image attribute labeling, using the CelebA dataset and the LLaVA-NeXT model. It evaluates AI-generated annotations on 1000 CelebA images across 40 binary attributes, incorporating a re-annotation step and majority voting to measure quality against human labels. Results show AI-human agreement of 79.5%, rising to 89.1% with re-annotations, with higher concordance on objective attributes, and AI labeling costing under 1% of manual annotation. The findings demonstrate potential for cost-effective AI annotations in specific tasks, while highlighting the need for scaling to more complex labels and addressing model biases.

Abstract

This study evaluates the capability of Vision-Language Models (VLMs) in image data annotation by comparing their performance on the CelebA dataset in terms of quality and cost-effectiveness against manual annotation. Annotations from the state-of-the-art LLaVA-NeXT model on 1000 CelebA images are in 79.5% agreement with the original human annotations. Incorporating re-annotations of disagreed cases into a majority vote boosts AI annotation consistency to 89.1% and even higher for more objective labels. Cost assessments demonstrate that AI annotation significantly reduces expenditures compared to traditional manual methods -- representing less than 1% of the costs for manual annotation in the CelebA dataset. These findings support the potential of VLMs as a viable, cost-effective alternative for specific annotation tasks, reducing both financial burden and ethical concerns associated with large-scale manual data annotation. The AI annotations and re-annotations utilized in this study are available on https://github.com/evev2024/EVEV2024_CelebA.

Paper Structure

This paper contains 8 sections, 2 equations, 1 figure, 5 tables.

Table of Contents

  1. Introduction
  2. Background
  3. Experiments
  4. Conclusions and Limitations
  5. Grouping of CelebA attributes
  6. Detailed comparison of AI annotation, original human annotation, and re-annotation
  7. Calculation of AI annotation costs
  8. Confidence intervals of the agreement percentages

Figures (1)

  • Figure 1: The performance of AI-generated annotations on the 40 CelebA attributes: (1) The more objective the attribute, the higher the agreement between AI and human annotations; (2) In cases of discrepancy, re-annotation aligned more with AI in exactly 50% of the attributes.