Are LLMs Ready for TOON? Benchmarking Structural Correctness-Sustainability Trade-offs in Novel Structured Output Formats

TL;DR

The paper tackles the environmental cost of structured-output generation by LLMs and introduces TOON, a compact data format, within a sustainability-aware benchmarking framework. It defines the Environment-Aware Generation Correctness Score $GCS_{env}$, combining structural correctness with carbon-aware efficiency to enable fair comparisons across formats. Through large-scale experiments across diverse models, TOON shows substantial token- and emission-efficiency gains, but often lags in strict structural correctness unless model capacity is high. The work demonstrates that format rankings depend on deployment priorities and underscores the value of environment-aware benchmarks for guiding practical, carbon-conscious LLM deployments.

Abstract

Large Language Models (LLMs) are increasingly required to generate structured, machine-readable outputs for downstream systems. While recent benchmarks have focused on evaluating the structural correctness of such outputs, the environmental impact of inference for different output formats has largely been overlooked. In this paper, we argue that structured output formats should be assessed not only in terms of correctness, but also with respect to their environmental efficiency. To this end, we introduce a sustainability-aware evaluation framework for structured generation that measures token usage, generation time, and estimated carbon emissions. Within this framework, we propose the Environment-Aware Generation Correctness Score (GCS_env), a unified metric that integrates structural correctness with carbon-aware efficiency. Using this framework, we systematically benchmark the novel TOON format against established representations (JSON, XML, YAML) across multiple LLMs spanning different architectures and parameter scales. Our results reveal a consistent trade-off: TOON yields markedly more compact outputs and lower emissions, but lower structural correctness when models lack native support. We show that increased model capacity reduces this gap and that environment-aware scoring can shift format rankings depending on deployment priorities. highlighting the need for sustainability-inclusive benchmarking and provides empirical evidence that compact representations such as TOON can offer practical advantages in large-scale, carbon-conscious LLM deployments.

Figures (4)

• Figure 1: Paired comparison between TOON and standard structured formats across correctness, efficiency, and environment-aware metrics.
• Figure 2: Titolo generale della griglia 3$\times$3.
• Figure 3: Token efficiency vs. carbon emissions across models and formats (e.g., $N_T$ vs. CE). Colors indicate formats; markers indicate models.
• Figure 4: Sensitivity of average scores to $\gamma \in [0,1]$. Each curve corresponds to a format (JSON, XML, YAML, and TOON variants).