SO-Bench: A Structural Output Evaluation of Multimodal LLMs

TL;DR

SO-Bench introduces a visual structured output benchmark to quantify how well multimodal LLMs generate JSON-schema-compliant outputs from images. It covers four visual domains and uses a large, multi-stage labeling pipeline to create 6.5K schemas and 1.8K image-schema pairs, with comprehensive AST-based evaluation metrics. Across open-source and frontier models, results show strong schema-adherence in top models but substantial gaps in field-level fidelity and full structural accuracy, especially for deeper schemas. Training with supervised fine-tuning and reinforcement learning with verifiable rewards substantially improves schema validation and field matching, indicating significant gains from targeted supervision. The work provides a community resource for benchmarking and improving multimodal structured reasoning.

Abstract

Multimodal large language models (MLLMs) are increasingly deployed in real-world, agentic settings where outputs must not only be correct, but also conform to predefined data schemas. Despite recent progress in structured generation in textual domain, there is still no benchmark that systematically evaluates schema-grounded information extraction and reasoning over visual inputs. In this work, we conduct a comprehensive study of visual structural output capabilities for MLLMs with our carefully designed SO-Bench benchmark. Covering four visual domains, including UI screens, natural images, documents, and charts, SO-Bench is built from over 6.5K diverse JSON schemas and 1.8K curated image-schema pairs with human-verified quality. Benchmarking experiments on open-sourced and frontier proprietary models reveal persistent gaps in predicting accurate, schema compliant outputs, highlighting the need for better multimodal structured reasoning. Beyond benchmarking, we further conduct training experiments to largely improve the model's structured output capability. We plan to make the benchmark available to the community.

Figures (15)

• Figure 1: An example of visual structured output task. Given a customized JSON schema often specified by the downstream applications, e.g. MenuReader, a model is tasked to extract information from input image, following the schema definition and user instruction.
• Figure 2: Overview of the multi-stage data generation pipeline for SO-Bench, including schema generation, user intent generation, and response generation stages. At each stage, we leverage proprietary frontier models (e.g., GPT-5 and Gemini-2.5-Pro) as generators with careful prompt design. Data from each stage is checked with human domain experts before passing to the next stage. Before the schema generation stage, input images and JSON schemas are embedded through a CLIP model for embedding search. Details of the pipeline is introduced in Section \ref{['sec:data_curation']}.
• Figure 3: (a).SO-Bench is collected from four categories: (1) UI: RICO deka2017rico, WebUI wu2023webui, ScreenSpot Pro li2025screenspot); (2) Documents: OmniDocBench ouyang2025omnidocbench, DocVQA mathew2021docvqa, InfographicVQA mathew2022infographicvqa; (3) Charts: ChartQA-Pro masry2025chartqapro, ChartMuseum tang2025chartmuseum; and (4) Natural images: HierText long2022towards. (b). Image feature distribution. We use CLIP radford2021learning image embedder to embed all images, and show their t-SNE features. The distribution from each image category is different, indicating the visual diversity.
• Figure 4: JSON schema data statistics in SO-Bench. The input JSON schemas are more complex than output formats in terms of the number of nested structure depths and the number of fields. Regarding image category, chart data shows the most complex schema structure.
• Figure 5: Top $100$ frequent feature counts in the input JSON schemas (best view with magnificence).