Improving Constrained Language Generation via Self-Distilled Twisted Sequential Monte Carlo

TL;DR

The paper tackles constrained language generation where the target distribution $\sigma(s_{1:T}|s_{0}) \propto p_{LM}(s_{1:T}|s_{0}) \phi(s_{1:T})$ is hard to sample from due to sparse rewards. It introduces self-distilled Twisted Sequential Monte Carlo, a two-phase loop that alternates self-distillation of the base LM with a modified contrastive twist learning using an generation-indexed effective potential $\phi^{(m)}$. The approach yields improved twist-induced proposals, as evidenced by decreasing $D_{KL}(\sigma \| q)$ across generations and better toxicity-diversity trade-offs than baselines, even with a small number of particles. This work offers a practical path to scalable, high-quality constrained generation in large language models and suggests directions for further efficiency and twist-learning enhancements.

Abstract

Recent work has framed constrained text generation with autoregressive language models as a probabilistic inference problem. Among these, Zhao et al. (2024) introduced a promising approach based on twisted Sequential Monte Carlo, which incorporates learned twist functions and twist-induced proposals to guide the generation process. However, in constrained generation settings where the target distribution concentrates on outputs that are unlikely under the base model, learning becomes challenging due to sparse and uninformative reward signals. We show that iteratively refining the base model through self-distillation alleviates this issue by making the model progressively more aligned with the target, leading to substantial gains in generation quality.

Figures (3)

• Figure 1: Similarity-Toxicity plot.
• Figure 2: Particle efficiency plot.
• Figure 3: Toxicity histograms.