Are Multimodal Large Language Models Good Annotators for Image Tagging?

TL;DR

TagLLM is proposed, a novel framework for image tagging that substantially narrows the gap between MLLM-generated and human annotations, especially in downstream training performance, where it closes about 60\% to 80\% of the difference.

Abstract

Image tagging, a fundamental vision task, traditionally relies on human-annotated datasets to train multi-label classifiers, which incurs significant labor and costs. While Multimodal Large Language Models (MLLMs) offer promising potential to automate annotation, their capability to replace human annotators remains underexplored. This paper aims to analyze the gap between MLLM-generated and human annotations and to propose an effective solution that enables MLLM-based annotation to replace manual labeling. Our analysis of MLLM annotations reveals that, under a conservative estimate, MLLMs can reduce annotation cost to as low as one-thousandth of the human cost, mainly accounting for GPU usage, which is nearly negligible compared to manual efforts. Their annotation quality reaches about 50\% to 80\% of human performance, while achieving over 90\% performance on downstream training tasks.Motivated by these findings, we propose TagLLM, a novel framework for image tagging, which aims to narrow the gap between MLLM-generated and human annotations. TagLLM comprises two components: Candidates generation, which employs structured group-wise prompting to efficiently produce a compact candidate set that covers as many true labels as possible while reducing subsequent annotation workload; and label disambiguation, which interactively calibrates the semantic concept of categories in the prompts and effectively refines the candidate labels. Extensive experiments show that TagLLM substantially narrows the gap between MLLM-generated and human annotations, especially in downstream training performance, where it closes about 60\% to 80\% of the difference.

Figures (7)

• Figure 1: An example of MLLM-generated annotations using different prompting methods. Green denotes true labels. Red denotes missing labels. Open-ended and multi-option prompting require only a single inference per image, whereas binary prompting requires $q$ inferences per image, where $q$ is the number of classes.
• Figure 2: The quality of annotations generated by Qwen3-VL-8B using different prompt formats and prompt styles.
• Figure 3: The quality of annotations and performance of training model obtained by using Qwen3-VL-8B.
• Figure 4: Comparison of three prompting strategies on COCO and Objects365. Left: Precision; middle: Recall; right: average number of candidate labels per image. DP (Disco-occurrence Partition) prompting yields the highest recall.
• Figure 5: Comparison of three prompting strategies on COCO and Objects365.