Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

POINTER: Constrained Progressive Text Generation via Insertion-based Generative Pre-training

Yizhe Zhang, Guoyin Wang, Chunyuan Li, Zhe Gan, Chris Brockett, Bill Dolan

TL;DR

POINTER introduces a novel insertion-based Transformer for hard-constrained text generation that progressively inserts tokens between existing ones across multiple stages, enabling a coarse-to-fine, non-autoregressive generation process. It combines stage-wise insertion with a special [NOI] token and an inner-layer beam search to maintain coherence while reducing time complexity. Pre-training on Wikipedia with BERT-style initialization and subsequent fine-tuning on News and Yelp demonstrate state-of-the-art performance on constrained-generation tasks, supported by both automatic metrics and human judgments. The work also provides DP-based data construction and a tailored decoding strategy, highlighting the practical potential of progressive insertion for controllable text synthesis.

Abstract

Large-scale pre-trained language models, such as BERT and GPT-2, have achieved excellent performance in language representation learning and free-form text generation. However, these models cannot be directly employed to generate text under specified lexical constraints. To address this challenge, we present POINTER (PrOgressive INsertion-based TransformER), a simple yet novel insertion-based approach for hard-constrained text generation. The proposed method operates by progressively inserting new tokens between existing tokens in a parallel manner. This procedure is recursively applied until a sequence is completed. The resulting coarse-to-fine hierarchy makes the generation process intuitive and interpretable. We pre-train our model with the proposed progressive insertion-based objective on a 12GB Wikipedia dataset, and fine-tune it on downstream hard-constrained generation tasks. Non-autoregressive decoding yields an empirically logarithmic time complexity during inference time. Experimental results on both News and Yelp datasets demonstrate that POINTER achieves state-of-the-art performance on constrained text generation. We released the pre-trained models and the source code to facilitate future research (https://github.com/dreasysnail/POINTER).

POINTER: Constrained Progressive Text Generation via Insertion-based Generative Pre-training

TL;DR

POINTER introduces a novel insertion-based Transformer for hard-constrained text generation that progressively inserts tokens between existing ones across multiple stages, enabling a coarse-to-fine, non-autoregressive generation process. It combines stage-wise insertion with a special [NOI] token and an inner-layer beam search to maintain coherence while reducing time complexity. Pre-training on Wikipedia with BERT-style initialization and subsequent fine-tuning on News and Yelp demonstrate state-of-the-art performance on constrained-generation tasks, supported by both automatic metrics and human judgments. The work also provides DP-based data construction and a tailored decoding strategy, highlighting the practical potential of progressive insertion for controllable text synthesis.

Abstract

Large-scale pre-trained language models, such as BERT and GPT-2, have achieved excellent performance in language representation learning and free-form text generation. However, these models cannot be directly employed to generate text under specified lexical constraints. To address this challenge, we present POINTER (PrOgressive INsertion-based TransformER), a simple yet novel insertion-based approach for hard-constrained text generation. The proposed method operates by progressively inserting new tokens between existing tokens in a parallel manner. This procedure is recursively applied until a sequence is completed. The resulting coarse-to-fine hierarchy makes the generation process intuitive and interpretable. We pre-train our model with the proposed progressive insertion-based objective on a 12GB Wikipedia dataset, and fine-tune it on downstream hard-constrained generation tasks. Non-autoregressive decoding yields an empirically logarithmic time complexity during inference time. Experimental results on both News and Yelp datasets demonstrate that POINTER achieves state-of-the-art performance on constrained text generation. We released the pre-trained models and the source code to facilitate future research (https://github.com/dreasysnail/POINTER).

Paper Structure

This paper contains 20 sections, 3 equations, 2 figures, 11 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Language Model Pre-training
  4. Method
  5. Model Overview
  6. Model Training
  7. Stage-wise Insertion Prediction
  8. Inference
  9. Experiments
  10. Experimental Setup
  11. Datasets and Pre-processing
  12. Experimental Results
  13. Conclusion
  14. Baseline and Experimental Details
  15. Additional Generated Examples for News Dataset
  16. ...and 5 more sections

Figures (2)

  • Figure 1: Illustration of the generation process ($X^0\to X$) of the proposed Pointer model. At each stage, the Insertion Transformer module generates either a regular token or a special $\mathtt{[NOI]}$ token for each gap between two existing tokens. The generation stops when all the gaps predict $\mathtt{[NOI]}$. The data preparation process reverses the above generative process.
  • Figure 2: Human evaluation template.