Composition-Grounded Instruction Synthesis for Visual Reasoning

TL;DR

COGS tackles the challenge of enabling modern multi-modal LLMs to perform complex reasoning in domains with scarce labeled data, such as charts and web GUIs. It introduces a three-stage, factor-based framework: decompose seed questions into primitive perception and reasoning factors, recombine these factors with new images to synthesize diverse question–answer pairs (along with factor-level subanswers), and fine-tune an MLLM via reinforcement learning using process-level rewards. The approach yields substantial gains in chart reasoning, generalizes across dataset mixtures, and extends effectively to webpage GUI reasoning, indicating strong transfer and generalization capabilities. By leveraging factor-level supervision and a compositional data-generation paradigm, COGS offers a scalable path to imbue MLLMs with robust, transferable reasoning skills for visually rich, structured content.

Abstract

Pretrained multi-modal large language models (MLLMs) demonstrate strong performance on diverse multimodal tasks, but remain limited in reasoning capabilities for domains where annotations are difficult to collect. In this work, we focus on artificial image domains such as charts, rendered documents, and webpages, which are abundant in practice yet lack large-scale human annotated reasoning datasets. We introduce COGS (COmposition-Grounded instruction Synthesis), a data-efficient framework for equipping MLLMs with advanced reasoning abilities from a small set of seed questions. The key idea is to decompose each seed question into primitive perception and reasoning factors, which can then be systematically recomposed with new images to generate large collections of synthetic question-answer pairs. Each generated question is paired with subquestions and intermediate answers, enabling reinforcement learning with factor-level process rewards. Experiments on chart reasoning show that COGS substantially improves performance on unseen questions, with the largest gains on reasoning-heavy and compositional questions. Moreover, training with a factor-level mixture of different seed data yields better transfer across multiple datasets, suggesting that COGS induces generalizable capabilities rather than dataset-specific overfitting. We further demonstrate that the framework extends beyond charts to other domains such as webpages.

Figures (10)

• Figure 1: COGS: Starting from a small set of reasoning-intensive seed questions, COGS decomposes them into primitive perception and reasoning factors, which are then recombined with new image sources to synthesize question–answer pairs. This process expands both the quantity and diversity of reasoning types beyond the original seeds. Fig. \ref{['fig:framework']} shows an illustrative example.
• Figure 2: The framework of COGS consists of three stages: seed question decomposition, factor recomposition, and model fine-tuning.
• Figure 3: Accuracy (%) on ChartQAPro by reasoning factor numbers and question types. COGS generally yields the larger gains on questions with more factors.
• Figure 4: Accuracy (%) on ChartQAPro by reasoning factor types with complexity from low to high.
• Figure 5: Qualitative evaluation examples. COGS-RL improved base models on questions that contain multiple factors and from different question types in ChartQAPro.

Theorems & Definitions (1)

• Proposition 3.1