FullAnno: A Data Engine for Enhancing Image Comprehension of MLLMs

TL;DR

The paper tackles the bottleneck of high-quality, fine-grained image-text data for training Multimodal Large Language Models (MLLMs), noting the scalability limits of GPT-4V-based data curation. It introduces FullAnno, a cascade data engine that leverages multiple expert models and rich prompts to automatically generate dense captions, region descriptions, OCR data, and object annotations, and applies this pipeline to re-annotate COCO and Visual Genome. The re-annotated dataset comprises 180k images with 4.16M object boxes, longer and more detailed captions (roughly 15× longer than original captions), and extensive OCR metadata, enabling richer image understanding. When the enhanced captions are used to pre-train LLaVA-v1.5-7B, substantial, consistent gains are observed across multiple vision-language benchmarks, validating the importance of dense, structured image annotations. The work provides a scalable data-generation solution and releases the regenerated COCO and Visual Genome data to support broader advancement in image-language grounding for ML systems.

Abstract

Multimodal Large Language Models (MLLMs) have shown promise in a broad range of vision-language tasks with their strong reasoning and generalization capabilities. However, they heavily depend on high-quality data in the Supervised Fine-Tuning (SFT) phase. The existing approaches aim to curate high-quality data via GPT-4V, but they are not scalable due to the commercial nature of GPT-4V and the simplicity of the prompts used to instruct the model. To this end, we devised the FullAnno system, which is a data engine that can generate large-scale, high-quality, and fine-grained image annotations consisting of the category and position of objects, region descriptions, text information, as well as image dense captions. This engine is characterized by its cascade annotation process, which involves multiple expert models and employs rich prompts to instruct LLMs in generating dense image captions. We re-annotated the COCO and Visual Genome datasets using our FullAnno system, tripling the number of object annotations and increasing the length of the original image captions by a factor of 15. Experiments show that the regenerated annotation can significantly enhance the capabilities of LLaVA-v1.5 on several benchmarks. The re-annotated data are available at: https://arcana-project-page.github.io

Figures (5)

• Figure 1: The Arcana data engine involves three crucial steps: (1) Augmenting and filtering image basic annotations, (2) obtaining text information and description for each annotated region, and (3) using a large language model to integrate these visual annotations into different types of captions.
• Figure 2: For each query, we illustrate the prompt construction process for instructing GPT-3.5. Note that "message" represents the final prompt. One example is displayed in the bottom row. The image is not visible to GPT-3.5 and is provided for reference only.
• Figure 3: Comparisons of dense caption between LLaVA-RaCap-COCO118k coco-llava and ours. The hallucination parts are highlighted in red, whereas detailed and accurate parts are emphasized in dark green.
• Figure 4: The region description includes various object attributes such as relative position, color, action, material, and emotion. Best viewed in color.
• Figure 5: The coarse-grained caption v.s. Our fine-grained caption used in pre-training stage of LLaVA. Important visual recognition-related description are highlighted in dark green.