ProVision: Programmatically Scaling Vision-centric Instruction Data for Multimodal Language Models

TL;DR

ProVision introduces a scalable, interpretable framework for vision-centric instruction data synthesis using scene graphs and programmable templates. By constructing augmented scene graphs for each image and leveraging 24 single-image plus 14 multi-image generators, ProVision builds ProVision-10M from Visual Genome and DataComp, enabling effective pre-training and instruction tuning of multimodal language models. Empirical results show consistent performance gains across a broad set of single-image and multi-image benchmarks, with dual-stage augmentation (pre-training and fine-tuning) and mixed data formats yielding the strongest improvements. The work demonstrates the practicality and cost-effectiveness of programmatic data generation for vision-language modeling, while acknowledging limitations around scene-graph quality and the potential for automated program synthesis to scale further.

Abstract

With the rise of multimodal applications, instruction data has become critical for training multimodal language models capable of understanding complex image-based queries. Existing practices rely on powerful but costly large language models (LLMs) or multimodal language models (MLMs) to produce instruction data. These are often prone to hallucinations, licensing issues and the generation process is often hard to scale and interpret. In this work, we present a programmatic approach that employs scene graphs as symbolic representations of images and human-written programs to systematically synthesize vision-centric instruction data. Our approach ensures the interpretability and controllability of the data generation process and scales efficiently while maintaining factual accuracy. By implementing a suite of 24 single-image, 14 multi-image instruction generators, and a scene graph generation pipeline, we build a scalable, cost-effective system: ProVision which produces diverse question-answer pairs concerning objects, attributes, relations, depth, etc., for any given image. Applied to Visual Genome and DataComp datasets, we generate over 10 million instruction data points, ProVision-10M, and leverage them in both pretraining and instruction tuning stages of MLMs. When adopted in the instruction tuning stage, our single-image instruction data yields up to a 7% improvement on the 2D split and 8% on the 3D split of CVBench, along with a 3% increase in performance on QBench2, RealWorldQA, and MMMU. Our multi-image instruction data leads to an 8% improvement on Mantis-Eval. Incorporation of our data in both pre-training and fine-tuning stages of xGen-MM-4B leads to an averaged improvement of 1.6% across 11 benchmarks.

Figures (5)

• Figure 1: Overview of ProVision system. It can generate a scene graph for any image, enabling programs to synthesize instruction data procedurally from the scene graph. If the scene graph exists, it directly generates instruction data based on it.
• Figure 2: We visualize the instruction data generation process and generated examples for both single and multiple image scenarios. For single image, instruction data can be categorized into six dimensions, aiming to improve the model's ability in retrieving and understanding the annotations. We divide instruction data into three categories for multiple images to help the model understand the relation between different images from the perspective of the relation between their scene graph.
• Figure 3: Our scene graph generation pipeline. For each input image, it will generate five annotations, including object, depth, segmentation, attribute, and relation, collectively forming an augmented scene graph.
• Figure 4: Results of instruction tuning LLaVA-1.5-7B with DC-S, i.e., single-image instruction data generated from the DataComp images and model-generated scene graphs.
• Figure 5: Results of instruction tuning Mantis-SigLIP-8B with DC-M, i.e., multi-image instruction data generated from the DataComp images and model-generated scene graphs.