Augmenting Image Annotation: A Human-LMM Collaborative Framework for Efficient Object Selection and Label Generation

TL;DR

This paper tackles the bottleneck of manual image annotation in computer vision due to labor intensity and annotator fatigue. It proposes a human–LMM collaborative framework that decouples object selection from label generation, with humans drawing bounding boxes and an LMM producing labels conditioned on image context, enabling bidirectional alignment. The approach is claimed to generalize across object recognition, scene description, and fine-grained categorization, and is validated with a case study on Asirra showing high accuracy and detailed breed-level labeling. The work suggests a scalable, efficient data labeling pipeline and discusses practical considerations, including data growth, costs, and the ethical impact on workers.

Abstract

Traditional image annotation tasks rely heavily on human effort for object selection and label assignment, making the process time-consuming and prone to decreased efficiency as annotators experience fatigue after extensive work. This paper introduces a novel framework that leverages the visual understanding capabilities of large multimodal models (LMMs), particularly GPT, to assist annotation workflows. In our proposed approach, human annotators focus on selecting objects via bounding boxes, while the LMM autonomously generates relevant labels. This human-AI collaborative framework enhances annotation efficiency by reducing the cognitive and time burden on human annotators. By analyzing the system's performance across various types of annotation tasks, we demonstrate its ability to generalize to tasks such as object recognition, scene description, and fine-grained categorization. Our proposed framework highlights the potential of this approach to redefine annotation workflows, offering a scalable and efficient solution for large-scale data labeling in computer vision. Finally, we discuss how integrating LMMs into the annotation pipeline can advance bidirectional human-AI alignment, as well as the challenges of alleviating the "endless annotation" burden in the face of information overload by shifting some of the work to AI.

Figures (3)

• Figure 1: Image Annotation Workflow. The figure illustrates the steps involved in the image annotation process. It begins with a collection of original images, followed by human selection of relevant images. Selected images are then annotated with bounding boxes to highlight objects of interest. The final output consists of labeled bounding boxes, which are used for downstream tasks in computer vision. The rightmost part indicates the task levels that annotators with different knowledge levels can complete. A single asterisk (*) marks labels that can be annotated by the average person. Double asterisks (**) mark labels that can be annotated by those with some foundational or passing-knowledge. Triple asterisks (***) signify labels that only expert groups are deemed capable of annotating (Note: GPT-4o can annotate at this level).
• Figure 2: A Synergistic Loop Illustrating Bidirectional Human–AI Alignment.
• Figure 3: A Synergistic Loop Illustrating Bidirectional Human–AI Alignment.