AutoTemplate: A Simple Recipe for Lexically Constrained Text Generation

TL;DR

AutoTemplate tackles lexically constrained text generation by decomposing the task into template generation and lexicalization, enabling the use of standard autoregressive encoder–decoder models with placeholders. The input is formed by prefixing constraint lexicons to the source, and the model learns to produce a template $\tilde{y}$ that aligns with placeholders, which are later lexicalized by substituting the actual constraints. Across keywords-to-sentence generation and entity-guided summarization, AutoTemplate achieves superior text quality while guaranteeing hard constraint satisfaction, outperforming both autoregressive and non-autoregressive baselines. The approach is simple, effective, and broadly applicable to tasks requiring precise constraint coverage without specialized decoding schemes.

Abstract

Lexically constrained text generation is one of the constrained text generation tasks, which aims to generate text that covers all the given constraint lexicons. While the existing approaches tackle this problem using a lexically constrained beam search algorithm or dedicated model using non-autoregressive decoding, there is a trade-off between the generated text quality and the hard constraint satisfaction. We introduce AutoTemplate, a simple yet effective lexically constrained text generation framework divided into template generation and lexicalization tasks. The template generation is to generate the text with the placeholders, and lexicalization replaces them into the constraint lexicons to perform lexically constrained text generation. We conducted the experiments on two tasks: keywords-to-sentence generations and entity-guided summarization. Experimental results show that the AutoTemplate outperforms the competitive baselines on both tasks while satisfying the hard lexical constraints. The code is available at https://github.com/megagonlabs/autotemplate

Figures (3)

• Figure 1: Illustration of AutoTemplate. We build the model input $\tilde{x}$ by concatenating the constraint lexicons $\mathcal{Z}$ with mask tokens. For the conditional text generation task, we further concatenate input document $x$. We also build the model output $\tilde{y}$ by masking the constraint lexicons in summary $y$. Then, we can train a standard sequence-to-sequence model, $p(\tilde{y} \mid \tilde{x})$, generate masked template $\tilde{y}$ given input $\tilde{x}$, and post-process to achieve lexically constrained text generation.
• Figure 2: Distribution of the number of oracle entities. The CNNDM dataset (left) tends to have longer summaries and contains more entities than the XSUM dataset. As the number of entities increases, it becomes more and more difficult to include all the entities in the generated summary.
• Figure 3: Success rate of entities included in the generated summary at a different number of entities. The green line denotes the BART model lewis-etal-2020-bart, the orange line denotes the CTRLSum model he-etal-2022-ctrlsum, and blue line denotes AutoTemplate model. These graphs show that CTRLSum can include a limited number of entities in summary with a high chance. However, it becomes more and more difficult as the number of entities increases, while AutoTemplate always satisfies the constraint.