GRAFT: GRaPH and Table Reasoning for Textual Alignment -- A Benchmark for Structured Instruction Following and Visual Reasoning

TL;DR

<3-5 sentence high-level summary> GRAFT introduces a synthetic, three-stage benchmark to probe instruction following and structured reasoning over charts and tables within multimodal models. It pairs programmatic data generation with a formal QA construction pipeline and a jury-based filtering process to produce schema-constrained, machine-parseable outputs (JSON/YAML), enabling precise evaluation of correctness, completeness, visual grounding, and format fidelity. The study demonstrates that while modern models can produce well-formed structured outputs, achieving high factual correctness and robust visual grounding—especially for table reasoning—remains challenging; larger, instruction-tuned, multimodal fusion models show the strongest performance. GRAFT provides a scalable, reproducible framework for diagnosing structured multimodal reasoning and guiding future improvements in visual–text alignment and reasoning.

Abstract

GRAFT is a structured multimodal benchmark designed to probe how well LLMs handle instruction following, visual reasoning, and tasks requiring tight visual textual alignment. The dataset is built around programmatically generated charts and synthetically rendered tables, each paired with a carefully constructed, multi step analytical question that depends solely on what can be inferred from the image itself. Responses are formatted in structured outputs such as JSON or YAML, enabling consistent and fine grained evaluation of both reasoning processes and adherence to output specifications. The benchmark further introduces a taxonomy of reasoning operations ranging from comparison and trend identification to ranking, aggregation, proportional estimation, and anomaly detection to support a comprehensive assessment of model capabilities. Taken together, GRAFT provides a unified and scalable framework for evaluating multimodal LLMs on visually grounded, structured reasoning tasks, offering a more rigorous standard for future benchmarking efforts.

Figures (13)

• Figure 1: GRAFT pipeline for synthetic visual generation and structured reasoning task construction.
• Figure 2: Pipeline for Model Evaluation.
• Figure 3: Comparative model performance on Chart-QnA and Table-QnA. Bars represent average scores across all evaluation axes.
• Figure :
• Figure :