Pix2Cap-COCO: Advancing Visual Comprehension via Pixel-Level Captioning

TL;DR

Pix2Cap-COCO introduces the first panoptic pixel-level caption dataset, enabling precise alignment between object masks and descriptive text to advance fine-grained visual-language understanding. The authors automate dataset creation with a GPT-4V–driven pipeline guided by Set-of-Mark and define panoptic segmentation-captioning as an end-to-end task, providing a baseline built on X-Decoder. They demonstrate the dataset's challenge and show meaningful gains when used to supervise fine-tuning of large multimodal models, notably on ViP-Bench and Visual Genome benchmarks. Overall, the work highlights the value of dense, per-object captions for grounding, segmentation, and language generation, with practical implications for fine-grained reasoning in multimodal systems.

Abstract

We present Pix2Cap-COCO, the first panoptic pixel-level caption dataset designed to advance fine-grained visual understanding. To achieve this, we carefully design an automated annotation pipeline that prompts GPT-4V to generate pixel-aligned, instance-specific captions for individual objects within images, enabling models to learn more granular relationships between objects and their contexts. This approach results in 167,254 detailed captions, with an average of 22.94 words per caption. Building on Pix2Cap-COCO, we introduce a novel task, panoptic segmentation-captioning, which challenges models to recognize instances in an image and provide detailed descriptions for each simultaneously. To benchmark this task, we design a robust baseline based on X-Decoder. The experimental results demonstrate that Pix2Cap-COCO is a particularly challenging dataset, as it requires models to excel in both fine-grained visual understanding and detailed language generation. Furthermore, we leverage Pix2Cap-COCO for Supervised Fine-Tuning (SFT) on large multimodal models (LMMs) to enhance their performance. For example, training with Pix2Cap-COCO significantly improves the performance of GPT4RoI, yielding gains in CIDEr +1.4%, ROUGE +0.4%, and SPICE +0.5% on Visual Genome dataset, and strengthens its region understanding ability on the ViP-BENCH, with an overall improvement of +5.1%, including notable increases in recognition accuracy +11.2% and language generation quality +22.2%.

Figures (7)

• Figure 1: Given a specified target (e.g., the second boy from the left), image-level caption and region-level caption both fail to align the visual input fully (indicates the misalignment). Meanwhile, a one-sentence caption is sometimes too short to describe the target, which leads to confusion (e.g., the boy marked with ). In contrast, a pixel-level caption can provide precise, detailed information that aligns accurately with the visual input.
• Figure 2: Overview of the Pix2Cap-COCO dataset pipeline. Step 1: Apply COCO mask annotations to the image; Step 2: Engineer prompts to generate detailed, formatted pixel-level captions; Step 3: Refine captions through hard matching and rephrasing with LLaMA; Step 4: Conduct manual re-annotation to establish an accurate benchmark.
• Figure 3: Analyses on Pix2Cap-COCO. Plot A shows the caption length distribution, with vertical dotted lines in the violin plot representing quartiles. We use $\log_{10}(\text{words per caption})$ as the horizontal axis to deal with the long-tail distribution of caption lengths. Plot B demonstrates the richness of attributes in our captions. Plot C is the distribution of the most common words in Pix2Cap-COCO captions.
• Figure 4: A visualization of Pix2Cap-COCO. Leveraging GPT-4V, our pixel-level captions contain comprehensive semantic information, including detailed object descriptions, interactions with surroundings, OCR information, and basic reasoning.
• Figure 5: An overview of our proposed model. Our proposed model follows an encoder-decoder architecture designed to perform the panoptic segmentation-captioning task end-to-end. Refer to the main text for details.