Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

World to Code: Multi-modal Data Generation via Self-Instructed Compositional Captioning and Filtering

Jiacong Wang, Bohong Wu, Haiyong Jiang, Xun Zhou, Xin Xiao, Haoyuan Guo, Jun Xiao

TL;DR

World to Code (W2C) presents a self-instructed, fully automated pipeline for generating high-quality multi-modal data by prompting Vision-Language Models to produce region-level captions and OCR data, followed by cross-prompt consistency filtering. The outputs are organized into Python code format to capture structured, executable visual information, reducing reliance on human annotation and specialized annotators. Empirical results show W2C improves a range of VQA and grounding benchmarks across multiple LLaVA backbones, and ablations demonstrate the benefits of code-format data and combined filtering. A key finding is that code-parsing outputs enable stronger few-shot performance than traditional detail captions, suggesting practical advantages for cross-modal understanding and downstream reasoning tasks.

Abstract

Recent advances in Vision-Language Models (VLMs) and the scarcity of high-quality multi-modal alignment data have inspired numerous researches on synthetic VLM data generation. The conventional norm in VLM data construction uses a mixture of specialists in caption and OCR, or stronger VLM APIs and expensive human annotation. In this paper, we present World to Code (W2C), a meticulously curated multi-modal data construction pipeline that organizes the final generation output into a Python code format. The pipeline leverages the VLM itself to extract cross-modal information via different prompts and filter the generated outputs again via a consistency filtering strategy. Experiments have demonstrated the high quality of W2C by improving various existing visual question answering and visual grounding benchmarks across different VLMs. Further analysis also demonstrates that the new code parsing ability of VLMs presents better cross-modal equivalence than the commonly used detail caption ability. Our code is available at https://github.com/foundation-multimodal-models/World2Code.

World to Code: Multi-modal Data Generation via Self-Instructed Compositional Captioning and Filtering

TL;DR

World to Code (W2C) presents a self-instructed, fully automated pipeline for generating high-quality multi-modal data by prompting Vision-Language Models to produce region-level captions and OCR data, followed by cross-prompt consistency filtering. The outputs are organized into Python code format to capture structured, executable visual information, reducing reliance on human annotation and specialized annotators. Empirical results show W2C improves a range of VQA and grounding benchmarks across multiple LLaVA backbones, and ablations demonstrate the benefits of code-format data and combined filtering. A key finding is that code-parsing outputs enable stronger few-shot performance than traditional detail captions, suggesting practical advantages for cross-modal understanding and downstream reasoning tasks.

Abstract

Recent advances in Vision-Language Models (VLMs) and the scarcity of high-quality multi-modal alignment data have inspired numerous researches on synthetic VLM data generation. The conventional norm in VLM data construction uses a mixture of specialists in caption and OCR, or stronger VLM APIs and expensive human annotation. In this paper, we present World to Code (W2C), a meticulously curated multi-modal data construction pipeline that organizes the final generation output into a Python code format. The pipeline leverages the VLM itself to extract cross-modal information via different prompts and filter the generated outputs again via a consistency filtering strategy. Experiments have demonstrated the high quality of W2C by improving various existing visual question answering and visual grounding benchmarks across different VLMs. Further analysis also demonstrates that the new code parsing ability of VLMs presents better cross-modal equivalence than the commonly used detail caption ability. Our code is available at https://github.com/foundation-multimodal-models/World2Code.
Paper Structure (43 sections, 6 equations, 4 figures, 9 tables, 1 algorithm)

This paper contains 43 sections, 6 equations, 4 figures, 9 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Vision Language Models
  4. Multi-modal Dataset Construction
  5. Code Representation for Visual Tasks
  6. Method
  7. Visual Concepts Extraction
  8. Self-Instructed Information Extraction
  9. Region-level Captions
  10. OCR information
  11. Information Filtering via Self Consistency
  12. Counting Filtering via Consistency
  13. Caption Re-ranking via Consistency
  14. Structured Formatting and Filtering
  15. Experiments
  16. ...and 28 more sections

Figures (4)

  • Figure 1: Overview of W2C and comparison of existing data construction pipelines. W2C differs from existing works by reducing the need for a mixture of specialists and expensive human annotations via self-instruct.
  • Figure 2: The data construction pipeline for W2C . Our pipeline utilizes both VLM and an object detector model to furnish structured data with region-specific awareness, detailed entity captions, and comprehensive global information. The VLM is iteratively invoked to generate the caption and perform consistency filtering to obtain high-quality data. The visual concepts set is obtained from the captions by the NLTK toolkit, $c_i$ here represents a visual concept from the set. The instruction prompts are all predefined templates.
  • Figure 3: Visualization of one W2C sample with OCR information.
  • Figure 4: Visualization of one W2C sample without OCR information.