FiLM: Fill-in Language Models for Any-Order Generation

TL;DR

FiLM introduces a fill-in language model that supports generation at any position by using Beta-distributed masking to blend bidirectional context into a decoder-only architecture. It provides a principled perplexity measure for any-order models and demonstrates strong text infilling and commonsense story completion, outperforming rearranged-data infillers and narrowing the gap with left-to-right CLMs as models scale. The approach is practical to fine-tune from existing MLMs or CLMs, offering a scalable path toward versatile bidirectional LLMs. Overall, FiLM presents a compelling, scalable alternative for flexible text generation and editing tasks that benefit from non-sequential generation orders.

Abstract

Language models have become the backbone of today's AI systems. However, their predominant left-to-right generation limits the use of bidirectional context, which is essential for tasks that involve filling text in the middle. We propose the Fill-in Language Model (FiLM), a new language modeling approach that allows for flexible generation at any position without adhering to a specific generation order. Its training extends the masked language modeling objective by adopting varying mask probabilities sampled from the Beta distribution to enhance the generative capabilities of FiLM. During inference, FiLM can seamlessly insert missing phrases, sentences, or paragraphs, ensuring that the outputs are fluent and are coherent with the surrounding context. In both automatic and human evaluations, FiLM outperforms existing infilling methods that rely on left-to-right language models trained on rearranged text segments. FiLM is easy to implement and can be either trained from scratch or fine-tuned from a left-to-right language model. Notably, as the model size grows, FiLM's perplexity approaches that of strong left-to-right language models of similar sizes, indicating FiLM's scalability and potential as a large language model.

Figures (12)

• Figure 1: Flexible sequence infilling by FiLM-1.6B. The given context is in black, and the text generated by the model is in color.
• Figure 2: Training and decoding of FiLM. During training, the mask probability $p$ is sampled according to a noise schedule, and then each token is independently replaced with [MASK] with probability $p$; FiLM is trained to predict the original tokens at the masked positions. At decoding time, the masks are sequentially filled in, each conditioned on the given context and previous predictions.
• Figure 3: Left: Illustration of the Beta distribution with varying modes. Right: Perplexity of FiLM when trained with different Beta distributions, each value depicted as the deviation from the baseline perplexity achieve by the uniform distribution.
• Figure 4: An illustration of FiLM decoded using min-entropy and max-entropy orders. The mask position selected to be filled in at each step is highlighted in the green color.
• Figure 5: Perplexity of FiLM and CLM on WikiText-103 (Left) and One Billion Word (Right).