StructuredRAG: JSON Response Formatting with Large Language Models

TL;DR

This work introduces StructuredRAG, a benchmark of six tasks designed to assess LLMs' proficiency in following response format instructions, and finds a high variance in performance across tasks, models, and prompting strategies with success rates ranging from 0 to 100%.

Abstract

The ability of Large Language Models (LLMs) to generate structured outputs, such as JSON, is crucial for their use in Compound AI Systems. However, evaluating and improving this capability remains challenging. In this work, we introduce StructuredRAG, a benchmark of six tasks designed to assess LLMs' proficiency in following response format instructions. We evaluate two state-of-the-art LLMs, Gemini 1.5 Pro and Llama 3 8B-instruct with 4-bit quantization using two distinct prompting strategies. We introduce these prompting strategies as f-String and Follow the Format (FF) prompting. Across 24 experiments, we find an average success rate of 82.55%. We further find a high variance in performance across tasks, models, and prompting strategies with success rates ranging from 0 to 100%. We find that Llama 3 8B-instruct often performs competitively with Gemini 1.5 Pro. We observe that task complexity significantly influences performance, with tasks involving lists or composite object outputs proving more challenging. Our findings highlight the need for further research into improving the reliability and consistency of structured output generation in LLMs. We have open-sourced our experimental code and results at github.com/weaviate/structured-rag.

Figures (6)

• Figure 1: An overview of our experimental results. Across 24 experiments, we achieve an average response format success rate of 82.55%. However, we find high variance in these results, 11 out of the 24 tests achieve 100% success, 2 out of 24 achieve 25% success or lower, and 5 of the tested methods achieve between 45% to 75% success..
• Figure 2: An illustration of the WikiQuestions dataset. Title-Abstract pairs are randomly sampled from Wikipedia. Gemini 1.5 Pro then synthesizes an answerable and unanswerable question for each. These generated questions are validated by a human annotator.
• Figure 3: A visualization of performance variance across all tasks from each model and prompting strategy tested.
• Figure 4: An illustration of performance variance across StructuredRAG tasks.
• Figure 5: An overview of the StructuredRAG benchmark. StructuredRAG tests response formatting across six different output type tests, string, integer, boolean, List[string], AnswerWithConfidence, and List[AnswerWithConfidence].