Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

StructEval: Benchmarking LLMs' Capabilities to Generate Structural Outputs

Jialin Yang, Dongfu Jiang, Lipeng He, Sherman Siu, Yuxuan Zhang, Disen Liao, Zhuofeng Li, Huaye Zeng, Yiming Jia, Haozhe Wang, Benjamin Schneider, Chi Ruan, Wentao Ma, Zhiheng Lyu, Yifei Wang, Yi Lu, Quy Duc Do, Ziyan Jiang, Ping Nie, Wenhu Chen

TL;DR

StructEval presents a comprehensive benchmark for assessing LLMs' ability to generate and convert highly structured outputs across 18 formats and 44 task types, spanning both non-renderable and renderable outputs. It introduces two complementary subsets (StructEval-T and StructEval-V) with a unified evaluation framework based on Syntax Score, Keyword Matching Score, and VQA Score, underpinned by a three-stage annotation pipeline to ensure high-quality data. The experimental results show a persistent gap between state-of-the-art commercial models and open-source alternatives, with generation tasks generally harder than conversions and renderable outputs posing greater challenges. By enabling automated, cross-format assessment and highlighting hard subtasks, StructEval aims to drive progress in robust, format-faithful structured output generation for real-world applications.

Abstract

As Large Language Models (LLMs) become integral to software development workflows, their ability to generate structured outputs has become critically important. We introduce StructEval, a comprehensive benchmark for evaluating LLMs' capabilities in producing both non-renderable (JSON, YAML, CSV) and renderable (HTML, React, SVG) structured formats. Unlike prior benchmarks, StructEval systematically evaluates structural fidelity across diverse formats through two paradigms: 1) generation tasks, producing structured output from natural language prompts, and 2) conversion tasks, translating between structured formats. Our benchmark encompasses 18 formats and 44 types of task, with novel metrics for format adherence and structural correctness. Results reveal significant performance gaps, even state-of-the-art models like o1-mini achieve only 75.58 average score, with open-source alternatives lagging approximately 10 points behind. We find generation tasks more challenging than conversion tasks, and producing correct visual content more difficult than generating text-only structures.

StructEval: Benchmarking LLMs' Capabilities to Generate Structural Outputs

TL;DR

StructEval presents a comprehensive benchmark for assessing LLMs' ability to generate and convert highly structured outputs across 18 formats and 44 task types, spanning both non-renderable and renderable outputs. It introduces two complementary subsets (StructEval-T and StructEval-V) with a unified evaluation framework based on Syntax Score, Keyword Matching Score, and VQA Score, underpinned by a three-stage annotation pipeline to ensure high-quality data. The experimental results show a persistent gap between state-of-the-art commercial models and open-source alternatives, with generation tasks generally harder than conversions and renderable outputs posing greater challenges. By enabling automated, cross-format assessment and highlighting hard subtasks, StructEval aims to drive progress in robust, format-faithful structured output generation for real-world applications.

Abstract

As Large Language Models (LLMs) become integral to software development workflows, their ability to generate structured outputs has become critically important. We introduce StructEval, a comprehensive benchmark for evaluating LLMs' capabilities in producing both non-renderable (JSON, YAML, CSV) and renderable (HTML, React, SVG) structured formats. Unlike prior benchmarks, StructEval systematically evaluates structural fidelity across diverse formats through two paradigms: 1) generation tasks, producing structured output from natural language prompts, and 2) conversion tasks, translating between structured formats. Our benchmark encompasses 18 formats and 44 types of task, with novel metrics for format adherence and structural correctness. Results reveal significant performance gaps, even state-of-the-art models like o1-mini achieve only 75.58 average score, with open-source alternatives lagging approximately 10 points behind. We find generation tasks more challenging than conversion tasks, and producing correct visual content more difficult than generating text-only structures.

Paper Structure

This paper contains 29 sections, 2 equations, 9 figures, 13 tables.

Table of Contents

  1. Introduction
  2. StructEval Dataset
  3. Annotation Pipeline
  4. Task Prompt
  5. Query/Metric Generation
  6. Expert Review
  7. StructEval Evaluation
  8. Syntax Score.
  9. Keyword Matching Score
  10. VQA Score
  11. Experiments
  12. Experimental Setup
  13. Evaluation Models.
  14. Inference Setup.
  15. Main Results
  16. ...and 14 more sections

Figures (9)

  • Figure 1: StructEval evaluates the LLM's capability to generate structured outputs, including text-only tasks like JSON, TOML, etc, and visual rendering tasks like HTML, React, Latex, etc.
  • Figure 2: The overall designed annotation pipeline of StructEval dataset
  • Figure 3: Example question and key words of the StructEval-T generation task
  • Figure 4: Example question, keywords, and VQA pairs for StructEval-V generation task
  • Figure 5: Prompt template used for VQA evaluation. We use GPT-4.1-mini in the benchmark evaluation.
  • ...and 4 more figures