Scaling Text-Rich Image Understanding via Code-Guided Synthetic Multimodal Data Generation

TL;DR

CoSyn presents a code-guided synthetic data framework to scale text-rich image understanding for vision-language models. By generating images from code via multiple renderers and using the code as context to produce instructional data, it builds CoSyn-400K (400K images, 2.7M instructions) and achieves state-of-the-art results on seven benchmarks among open-source models, while enabling strong zero-shot and domain-adaptation performance (e.g., NutritionQA) with minimal in-domain data. The approach also provides synthetic pointing data to ground information in images, enhancing agent-like capabilities for real-world tasks. Overall, CoSyn demonstrates that structured synthetic data, guided by LLMs and diverse rendering pipelines, can substantially improve generalization and data efficiency in text-rich multimodal understanding.

Abstract

Reasoning about images with rich text, such as charts and documents, is a critical application of vision-language models (VLMs). However, VLMs often struggle in these domains due to the scarcity of diverse text-rich vision-language data. To address this challenge, we present CoSyn, a framework that leverages the coding capabilities of text-only large language models (LLMs) to automatically create synthetic text-rich multimodal data. Given input text describing a target domain (e.g., "nutrition fact labels"), CoSyn prompts an LLM to generate code (Python, HTML, LaTeX, etc.) for rendering synthetic images. With the underlying code as textual representations of the synthetic images, CoSyn can generate high-quality instruction-tuning data, again relying on a text-only LLM. Using CoSyn, we constructed a dataset comprising 400K images and 2.7M rows of vision-language instruction-tuning data. Comprehensive experiments on seven benchmarks demonstrate that models trained on our synthetic data achieve state-of-the-art performance among competitive open-source models, including Llama 3.2, and surpass proprietary models such as GPT-4V and Gemini 1.5 Flash. Furthermore, CoSyn can produce synthetic pointing data, enabling VLMs to ground information within input images, showcasing its potential for developing multimodal agents capable of acting in real-world environments.

Figures (19)

• Figure 1: Given a novel task (e.g., answering questions about nutrition facts), our code-guided generation system can produce targeted synthetic data to enhance the performance of VLMs on that specific task.
• Figure 2: The overview of our Code Guided Synthetic data generation system (CoSyn), which has 20 generation pipelines based on 11 render tools. Given a user query, e.g., "book cover," CoSyn selects the appropriate pipelines and starts by generating diverse topics conditioned on personas, then synthesizes detailed data for code generation. The code renders the image and is also fed as context for an LLM to construct instruction-tuning data.
• Figure 3: Our CoSyn-400K dataset consists of 9 categories of text-rich images with 2.7M instruction-tuning data. More qualitative examples, along with question-answer annotations, are available in Figure \ref{['fig: chart_example']} -\ref{['fig: special_example']} in Appendix \ref{['appendix: example']}.
• Figure 4: Ablation on training data selection. Aux, Syn, and Eval stand for auxiliary, synthetic, and evaluation datasets, respectively. We report the average score on seven benchmarks. The detailed performance breakdown on each benchmark is in Table \ref{['tab:train_data_ablation']}.
• Figure 5: Zero shot performance on NutritionQA. The x-axis denotes the number of training examples used for the instruction-tuning stage. The models on the upper left side demonstrate better data efficiency.