Meaning Typed Prompting: A Technique for Efficient, Reliable Structured Output Generation

TL;DR

This work introduces Meaning Typed Prompting (MTP), a technique for efficient structured output generation that integrates types, meanings, and abstractions, such as variables and classes, into the prompting process, and enhances output clarity and reduces dependence on complex abstractions.

Abstract

Extending Large Language Models (LLMs) to advanced applications requires reliable structured output generation. Existing methods which often rely on rigid JSON schemas, can lead to unreliable outputs, diminished reasoning capabilities, and increased computational overhead, limiting LLMs' adaptability for complex tasks. We introduce Meaning Typed Prompting (MTP), a technique for efficient structured output generation that integrates types, meanings, and abstractions, such as variables and classes, into the prompting process. By utilizing expressive type definitions, MTP enhances output clarity and reduces dependence on complex abstractions, simplifying development, and improving implementation efficiency. This enables LLMs to understand relationships and generate structured data more effectively. Empirical evaluations on multiple benchmarks demonstrate that MTP outperforms existing frameworks in accuracy, reliability, consistency, and token efficiency. We present Semantix, a framework that implements MTP, providing practical insights into its application.

Figures (24)

• Figure 1: Comparison of structured output methods—OpenAI, Semantix, and DSPy—using identical code constructs and the same LLM (gpt-4o-mini). OpenAI and DSPy convert types into JSON Schema and output JSON; Semantix uses natural language representations and outputs object representations. Semantix has the lowest input token usage and, for longer outputs, consistently uses fewer tokens due to less frequent use of {}".
• Figure 2: Structure of the Meaning Typed Prompt and example Final Prompt for generating a Person object for a given name with attributes as first name, last name, year of birth and personality as an enumerated type.
• Figure 3: Semantix merges Types, Variables, Functions, and the Main Function into an enhanced function which generates MTP at runtime. *Type Hints can be extended using semantic types.
• Figure 4: Execution of an Enhanced Function in Semantix includes querying the LLM, transforming the output into code, and managing errors.
• Figure 5: Reliability and performance metrics and token usage for structured output generation benchmarks with 0 retries. Both Semantix and Fructose performed exceptionally well across all three benchmark tasks, but Semantix demonstrated significantly lower token usage.