NeuroLogic Decoding: (Un)supervised Neural Text Generation with Predicate Logic Constraints

TL;DR

NeuroLogic Decoding tackles the challenge of generating text under complex lexical constraints by integrating predicate-logic constraints into decoding. It represents constraints in CNF, tracks clause states with dedicated prefix tries, and performs pruning, grouping, and selection to search for high-likelihood outputs that satisfy all constraints with beam-search-like efficiency. Across commonsense generation, recipe writing, data-grounded dialogue, and translation bias mitigation, NeuroLogic yields consistent gains in constraint satisfaction and generation quality, including strong zero-shot performance and notable improvements over supervised baselines. The work suggests inference-time constraint enforcement can rival or exceed the benefits of large-scale finetuning, offering a practical path to controllable generation without architectural changes.

Abstract

Conditional text generation often requires lexical constraints, i.e., which words should or shouldn't be included in the output text. While the dominant recipe for conditional text generation has been large-scale pretrained language models that are finetuned on the task-specific training data, such models do not learn to follow the underlying constraints reliably, even when supervised with large amounts of task-specific examples. We propose NeuroLogic Decoding, a simple yet effective algorithm that enables neural language models -- supervised or not -- to generate fluent text while satisfying complex lexical constraints. Our approach is powerful yet efficient. It handles any set of lexical constraints that is expressible under predicate logic, while its asymptotic runtime is equivalent to conventional beam search. Empirical results on four benchmarks show that NeuroLogic Decoding outperforms previous approaches, including algorithms that handle a subset of our constraints. Moreover, we find that unsupervised models with NeuroLogic Decoding often outperform supervised models with conventional decoding, even when the latter is based on considerably larger networks. Our results suggest the limit of large-scale neural networks for fine-grained controllable generation and the promise of inference-time algorithms.

Figures (5)

• Figure 1: Overview of several constrained generation tasks. For instance, generating a short description from a set of concepts lin2019commongen requires using each of those words at least once; this can be expressed as a logical expression (here, '($\textrm{food} \lor \textrm{foods})\land\ldots$'). Our proposed NeuroLogic Decoding handles all predicate logic constraints efficiently, yet with the same asymptotic runtime as beam search.
• Figure 2: Clause states and possible transitions. $D_i$ and $\neg D_i$ denote positive and negative literal respectively.
• Figure 3: Illustration of the NeuroLogic decoding procedure. In this example, $k=3$, $\alpha=8$, $\beta=2$, $\lambda=0.1$
• Figure 4: Performance (y-axis) of supervised GPT2-Large on CommonGen, with a varying amount of training data for supervision (x-axis). The orange line denotes decoding with NeuroLogic, and the blue line denotes decoding with conventional beam search.
• Figure 6: Generation examples of different models in supervised and zero-shot setting with and without NeuroLogic Decoding, on CommonGen.