Efficient Guided Generation for Large Language Models
Brandon T. Willard, Rémi Louf
TL;DR
The paper tackles the challenge of guiding large language model text generation to conform to rigid structural constraints (regular expressions and CFGs) with high efficiency. It reframes constraint satisfaction as finite-state machine processing and introduces a vocabulary-indexing technique that enables O(1) average masking per step, with extensions to pushdown automata for CFG parsing. The approach, implemented in the Outlines library and demonstrated against baselines like Guidance, achieves substantial speedups and modest memory usage while preserving flexibility and model-agnostic applicability. The work suggests promising avenues for constrained decoding, training-time benefits, and potential integration of masking directly into model architectures for further efficiency gains.
Abstract
In this article we show how the problem of neural text generation can be constructively reformulated in terms of transitions between the states of a finite-state machine. This framework leads to an efficient approach to guiding text generation with regular expressions and context-free grammars by allowing the construction of an index over a language model's vocabulary. The approach is model agnostic, allows one to enforce domain-specific knowledge and constraints, and enables the construction of reliable interfaces by guaranteeing the structure of the generated text. It adds little overhead to the token sequence generation process and significantly outperforms existing solutions. An implementation is provided in the open source Python library Outlines