JSONSchemaBench: A Rigorous Benchmark of Structured Outputs for Language Models

TL;DR

The paper presents JSONSchemaBench, a large-scale benchmark of 10K real-world JSON schemas to evaluate constrained decoding for structured outputs. It defines an executable, three-dimensional evaluation framework focusing on efficiency, coverage, and quality, and assesses six leading constrained-decoding frameworks against both empirical schema compliance and the JSON Schema Test Suite. The findings show constrained decoding can significantly speed generation, with varying coverage across frameworks and substantial, typically modest, improvements in downstream task accuracy, especially when using Guidance. The work establishes a practical standard for evaluating constrained decoding and offers actionable insights for improving structured generation in real-world applications.

Abstract

Reliably generating structured outputs has become a critical capability for modern language model (LM) applications. Constrained decoding has emerged as the dominant technology across sectors for enforcing structured outputs during generation. Despite its growing adoption, little has been done with the systematic evaluation of the behaviors and performance of constrained decoding. Constrained decoding frameworks have standardized around JSON Schema as a structured data format, with most uses guaranteeing constraint compliance given a schema. However, there is poor understanding of the effectiveness of the methods in practice. We present an evaluation framework to assess constrained decoding approaches across three critical dimensions: efficiency in generating constraint-compliant outputs, coverage of diverse constraint types, and quality of the generated outputs. To facilitate this evaluation, we introduce JSONSchemaBench, a benchmark for constrained decoding comprising 10K real-world JSON schemas that encompass a wide range of constraints with varying complexity. We pair the benchmark with the existing official JSON Schema Test Suite and evaluate six state-of-the-art constrained decoding frameworks, including Guidance, Outlines, Llamacpp, XGrammar, OpenAI, and Gemini. Through extensive experiments, we gain insights into the capabilities and limitations of constrained decoding on structured generation with real-world JSON schemas. Our work provides actionable insights for improving constrained decoding frameworks and structured generation tasks, setting a new standard for evaluating constrained decoding and structured generation. We release JSONSchemaBench at https://github.com/guidance-ai/jsonschemabench

Figures (9)

• Figure 1: Comparison across various constrained-decoding frameworks by efficiency (speed of output generation), coverage (support for JSON Schema features), and quality (effects on underlying task accuracy). Guidance outperforms other frameworks on these dimensions.
• Figure 2: Feature and Format constraint distribution.
• Figure 3: Prompt template used to generate JSON objects in the coverage experiment.
• Figure 4: Feature checklist for different structured output engines
• Figure 5: Illustration of over-constrained and under-constrained.

Theorems & Definitions (6)

• Definition 5.1: Declared Coverage
• Definition 5.2: Empirical Coverage
• Definition 5.3: True Coverage
• Definition 5.4: Over-constrained
• Definition 5.5: Under-constrained
• Definition C.1: Theoretical Coverage