PARSE: LLM Driven Schema Optimization for Reliable Entity Extraction

TL;DR

PARSE reframes JSON schemas as optimizable interfaces rather than fixed contracts, addressing reliability gaps in LLM-driven structured extraction for Software 3.0. It combines ARCHITECT, which iteratively refines schemas (while preserving compatibility via RELAY), with SCOPE, a reflection-based extractor that enforces multi-stage guardrails. Across SGD, SWDE, and Retail datasets, PARSE achieves substantial accuracy gains (up to 64.7% on SWDE) and strong error reduction (≈92% on first retries), albeit with some latency costs mitigated by optimized schemas. This work demonstrates a practical, co-optimization approach to enable reliable, scalable LLM agent systems that can robustly extract structured data from diverse unstructured sources.

Abstract

Structured information extraction from unstructured text is critical for emerging Software 3.0 systems where LLM agents autonomously interact with APIs and tools. Recent approaches apply large language models directly to extraction tasks using existing JSON schemas, often with constraint decoding or reinforcement learning approaches to ensure syntactic validity, but treat JSON schemas as static contracts designed for human developers, leading to suboptimal extraction performance, frequent hallucinations, and unreliable agent behavior when schemas contain ambiguous or incomplete specifications. We recognize that JSON schemas themselves are a form of natural language understanding contract that encodes rules, relationships, and expectations about data structure contracts that LLMs should be able to both interpret and systematically improve. Consequently, we develop PARSE (Parameter Automated Refinement and Schema Extraction), a novel system with two synergistic components: ARCHITECT, which autonomously optimizes JSON schemas for LLM consumption while maintaining backward compatibility through RELAY (an integrated code generation system), and SCOPE, which implements reflection-based extraction with combined static and LLM-based guardrails. We evaluate PARSE qualitatively and quantitatively on three datasets including Schema-Guided Dialogue (SGD), Structured Web Data Extraction (SWDE), and internal retail conversation data, and find that it achieves up to 64.7% improvement in extraction accuracy on SWDE with combined framework improvements reaching 10% across models, while reducing extraction errors by 92% within the first retry and and maintaining practical latency.

Figures (5)

• Figure 1: Overview of PARSE framework showing schema optimization and extraction pipeline. The system takes an unoptimized schema and input text (left), processes it through ARCHITECT for schema refinement and SCOPE for extraction (center), producing accurate structured output (right). The diagram highlights how PARSE outperforms other information extraction agents by implementing robust schema optimization and validation. Numbers indicate the sequential flow of operations.
• Figure 2: Detailed architecture of PARSE's two main components: ARCHITECT and SCOPE. The Build Phase (top) shows ARCHITECT's workflow for schema optimization through iterative refinement using synthetic test data generation and validation. The Extract Phase (bottom) illustrates SCOPE's extraction pipeline with built-in guardrails and reflection mechanisms for ensuring reliable structured output. Numbers indicate the sequential flow of operations.
• Figure 3: Error recovery with and without reflection and with and without ARCHITECT schemas
• Figure 4: Performance of schemas on validation sets over ARCHITECT optimisation iterations
• Figure 5: Description of how frequently different types of changes are done through ARCHITECT to optimise a schema