SLOT: Structuring the Output of Large Language Models

TL;DR

SLOT tackles the challenge of producing reliable structured outputs from LLMs by introducing a model-agnostic post-processing framework that maps unstructured text to a predefined JSON schema using a lightweight, fine-tuned model. It pairs this with a synthetic data pipeline and a formal evaluation framework based on schema accuracy and content similarity, demonstrating that small open-weight models with SFT can outperform larger proprietary models in structured generation. The results reveal strong gains, especially when SLOT is combined with constrained decoding, achieving near-perfect schema conformity and high semantic fidelity across diverse datasets. This approach significantly improves the practicality and accessibility of structured generation for real-world LLM applications in resource-constrained settings.

Abstract

Structured outputs are essential for large language models (LLMs) in critical applications like agents and information extraction. Despite their capabilities, LLMs often generate outputs that deviate from predefined schemas, significantly hampering reliable application development. We present SLOT (Structured LLM Output Transformer), a model-agnostic approach that transforms unstructured LLM outputs into precise structured formats. While existing solutions predominantly rely on constrained decoding techniques or are tightly coupled with specific models, SLOT employs a fine-tuned lightweight language model as a post-processing layer, achieving flexibility across various LLMs and schema specifications. We introduce a systematic pipeline for data curation and synthesis alongside a formal evaluation methodology that quantifies both schema accuracy and content fidelity. Our results demonstrate that fine-tuned Mistral-7B model with constrained decoding achieves near perfect schema accuracy (99.5%) and content similarity (94.0%), outperforming Claude-3.5-Sonnet by substantial margins (+25 and +20 percentage points, respectively). Notably, even compact models like Llama-3.2-1B can match or exceed the structured output capabilities of much larger proprietary models when equipped with SLOT, enabling reliable structured generation in resource-constrained environments.

Figures (8)

• Figure 1: SLOT converts a textual LLM response into structured JSON with a pre-defined schema.
• Figure 2: Top: Statistics of the datasets used in experiments. $^{\diamond}$ refers to partially synthesized for repurposing and $^{\star}$ refers to fully synthesized. Bottom: JSON complexity in different dimensions.
• Figure 3: Our proposed framework for post-training and hosting an LLM for structured output generation.
• Figure 4: Approaches for LLM structured outputs. a) prompting LLM form structured output, b) constrained decoding, c) post-training, and d) SLOT.
• Figure 5: Data curation pipeline for synthetic training data and partially synthetic test data.