Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Text Infilling

Wanrong Zhu, Zhiting Hu, Eric Xing

TL;DR

The paper tackles the general task of text infilling with multiple, potentially long blanks in a template. It proposes a Transformer-based self-attention model with segment-aware positional encoding to capture global and surrounding context, and compares it against seq2seq and GAN baselines. Through extensive supervised experiments on Yelp, Grimm tales, and NBA data, the self-attention approach consistently yields higher automatic and human-evaluated quality, establishing a strong baseline for future research. The work also discusses connections to pre-trained language models like BERT and outlines avenues for leveraging such representations in infilling tasks.

Abstract

Recent years have seen remarkable progress of text generation in different contexts, such as the most common setting of generating text from scratch, and the emerging paradigm of retrieval-and-rewriting. Text infilling, which fills missing text portions of a sentence or paragraph, is also of numerous use in real life, yet is under-explored. Previous work has focused on restricted settings by either assuming single word per missing portion or limiting to a single missing portion to the end of the text. This paper studies the general task of text infilling, where the input text can have an arbitrary number of portions to be filled, each of which may require an arbitrary unknown number of tokens. We study various approaches for the task, including a self-attention model with segment-aware position encoding and bidirectional context modeling. We create extensive supervised data by masking out text with varying strategies. Experiments show the self-attention model greatly outperforms others, creating a strong baseline for future research.

Text Infilling

TL;DR

The paper tackles the general task of text infilling with multiple, potentially long blanks in a template. It proposes a Transformer-based self-attention model with segment-aware positional encoding to capture global and surrounding context, and compares it against seq2seq and GAN baselines. Through extensive supervised experiments on Yelp, Grimm tales, and NBA data, the self-attention approach consistently yields higher automatic and human-evaluated quality, establishing a strong baseline for future research. The work also discusses connections to pre-trained language models like BERT and outlines avenues for leveraging such representations in infilling tasks.

Abstract

Recent years have seen remarkable progress of text generation in different contexts, such as the most common setting of generating text from scratch, and the emerging paradigm of retrieval-and-rewriting. Text infilling, which fills missing text portions of a sentence or paragraph, is also of numerous use in real life, yet is under-explored. Previous work has focused on restricted settings by either assuming single word per missing portion or limiting to a single missing portion to the end of the text. This paper studies the general task of text infilling, where the input text can have an arbitrary number of portions to be filled, each of which may require an arbitrary unknown number of tokens. We study various approaches for the task, including a self-attention model with segment-aware position encoding and bidirectional context modeling. We create extensive supervised data by masking out text with varying strategies. Experiments show the self-attention model greatly outperforms others, creating a strong baseline for future research.

Paper Structure

This paper contains 36 sections, 2 equations, 2 figures, 8 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Text Infilling
  4. Problem Definition
  5. Preliminary Solutions
  6. Experiments
  7. Varying Mask Rates and #Blanks
  8. Quantitative and Human Evaluation
  9. Samples
  10. Long Content Infilling
  11. Quantitative and Human Evaluation
  12. Samples
  13. Conclusion
  14. More Details of Text Infilling Self-Attention Model
  15. Notations
  16. ...and 21 more sections

Figures (2)

  • Figure 1: An example of text infilling.
  • Figure 2: The overall structure of Self-attn. This figure depicts the training process. The decoder will attend to the template at each position, conditioning on the template together with what has been filled in the template. During inference, the input will not go through the masked multi-head attention layer.