Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Thinking Before Constraining: A Unified Decoding Framework for Large Language Models

Ngoc Trinh Hung Nguyen, Alonso Silva, Laith Zumot, Liubov Tupikina, Armen Aghasaryan, Mehwish Alam

TL;DR

This work introduces In-Writing, a hybrid decoding framework that merges natural language reasoning with structured, constraint-based output to produce reliable JSON-like results. By allowing unconstrained reasoning until a trigger token is emitted, then switching to regex- or FSM-guided constrained decoding, the method preserves expressive reasoning while ensuring parseable final outputs. Across diverse reasoning and classification datasets, In-Writing achieves up to $27\%$ accuracy gains with a modest token overhead of about $10$-$20$ tokens and demonstrates applicability to smaller models, offering practical benefits for industrial tasks requiring reliable structured outputs. Limitations include potential premature trigger generation and looping, motivating future work on improved trigger design and prompt strategies.

Abstract

Natural generation allows Language Models (LMs) to produce free-form responses with rich reasoning, but the lack of guaranteed structure makes outputs difficult to parse or verify. Structured generation, or constrained decoding, addresses this drawback by producing content in standardized formats such as JSON, ensuring consistency and guaranteed-parsable outputs, but it can inadvertently restrict the model's reasoning capabilities. In this work, we propose a simple approach that combines the advantages of both natural and structured generation. By allowing LLMs to reason freely until specific trigger tokens are generated, and then switching to structured generation, our method preserves the expressive power of natural language reasoning while ensuring the reliability of structured outputs. We further evaluate our approach on several datasets, covering both classification and reasoning tasks, to demonstrate its effectiveness, achieving a substantial gain of up to 27% in accuracy compared to natural generation, while requiring only a small overhead of 10-20 extra tokens.

Thinking Before Constraining: A Unified Decoding Framework for Large Language Models

TL;DR

This work introduces In-Writing, a hybrid decoding framework that merges natural language reasoning with structured, constraint-based output to produce reliable JSON-like results. By allowing unconstrained reasoning until a trigger token is emitted, then switching to regex- or FSM-guided constrained decoding, the method preserves expressive reasoning while ensuring parseable final outputs. Across diverse reasoning and classification datasets, In-Writing achieves up to accuracy gains with a modest token overhead of about - tokens and demonstrates applicability to smaller models, offering practical benefits for industrial tasks requiring reliable structured outputs. Limitations include potential premature trigger generation and looping, motivating future work on improved trigger design and prompt strategies.

Abstract

Natural generation allows Language Models (LMs) to produce free-form responses with rich reasoning, but the lack of guaranteed structure makes outputs difficult to parse or verify. Structured generation, or constrained decoding, addresses this drawback by producing content in standardized formats such as JSON, ensuring consistency and guaranteed-parsable outputs, but it can inadvertently restrict the model's reasoning capabilities. In this work, we propose a simple approach that combines the advantages of both natural and structured generation. By allowing LLMs to reason freely until specific trigger tokens are generated, and then switching to structured generation, our method preserves the expressive power of natural language reasoning while ensuring the reliability of structured outputs. We further evaluate our approach on several datasets, covering both classification and reasoning tasks, to demonstrate its effectiveness, achieving a substantial gain of up to 27% in accuracy compared to natural generation, while requiring only a small overhead of 10-20 extra tokens.
Paper Structure (26 sections, 2 figures, 8 tables, 2 algorithms)

This paper contains 26 sections, 2 figures, 8 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. Related Work
  4. In-Writting Method
  5. Experiments
  6. Datasets
  7. Reasoning Tasks
  8. Classification Tasks
  9. Evaluation Metrics
  10. Prompts and Models
  11. Output Schema
  12. Implementation Details
  13. Results and Discussion
  14. Structured Generation Is an Effective Parser
  15. Impact of Formats on Structured Generation Performance
  16. ...and 11 more sections

Figures (2)

  • Figure 1: Comparison between Natural Generation and the proposed In-Writing method on Llama3-8B-it. Both produce identical reasoning; In-Writing additionally generates an easily parsable final answer with syntactic corrections (e.g., directing Helga to choice A).
  • Figure 2: Illustrative example of the In-Writing approach. The model first generates unconstrained reasoning and then switches to guided decoding (state $0$) once a trigger token is produced.

Theorems & Definitions (1)

  • Definition 1: Finite Automaton