Sentence Curve Language Models

TL;DR

This work addresses the limitation of static target embeddings in diffusion and non-autoregressive language models by introducing sentence curves, a spline-based continuous representation of sentences. The Sentence Curve Language Model (SCLM) predicts target sentence curves via control points, providing a global-structure regularization that complements local word-level prediction. The authors establish theoretical insights showing how sentence curve prediction promotes sentence-level coherence and present a $K$-curve extension to alleviate multimodality. Empirically, SCLMs achieve state-of-the-art results among diffusion LMs on IWSLT14 and WMT14, exhibit stable training without sequence-level knowledge distillation, and show promising language modeling results on LM1B, with stronger sentence-level correlations observed in semi-AR settings.

Abstract

Language models (LMs) are a central component of modern AI systems, and diffusion-based language models (DLMs) have recently emerged as a competitive alternative. Both paradigms rely on word embeddings not only to represent the input sentence, but also to represent the target sentence that backbone models are trained to predict. We argue that such static embedding of the target word is insensitive to neighboring words, encouraging locally accurate word prediction while neglecting global structure across the target sentence. To address this limitation, we propose a continuous sentence representation, termed sentence curve, defined as a spline curve whose control points affect multiple words in the sentence. Based on this representation, we introduce sentence curve language model (SCLM), which extends DLMs to predict sentence curves instead of the static word embeddings. We theoretically show that sentence curve prediction induces a regularization effect that promotes global structure modeling, and characterize how different sentence curve types affect this behavior. Empirically, SCLM achieves SOTA performance among DLMs on IWSLT14 and WMT14, shows stable training without burdensome knowledge distillation, and demonstrates promising potential compared to discrete DLMs on LM1B.

Figures (9)

• Figure 1: (Top) Sentence curve generation example of our SCLM in the denoising steps. More examples are illustrated in Figure \ref{['fig:extra_curve_generation_examples']} in Appendix \ref{['appendix:sentence_curve_generation_examples']}. (Bottom) Performance history of DLMs in IWSLT14 De$\rightarrow$En (blue) and WMT14 En$\rightarrow$De (orange).
• Figure 2: (Left) Overview of our SCLM's training process at noising step $t$ (Section \ref{['subsec:sentence_curve_language_model']}). Starting from 'Target Sentence' and its 'Target Embeddings', noise is added to obtain 'Noised Embeddings'. Then, it is mapped to sentence curve (noised), $[\mathbf{p}_i^t]_{i=1}^L$, and fed into the model. The model predicts a denoised sentence curve $[\mathbf{\hat{p}}_i^0]_{i=1}^L$, which is mapped back to 'Denoised Embeddings' for loss computation. (Right) Illustration of the $K$-sentence curve prediction scheme.
• Figure 3: Illustration of control points and embeddings for the sentence “I love my dog” with $\eta=3$. Smaller red dots denote control points ($\mathbf{p}$), and larger dots denote word embeddings ($\mathbf{e}$). The black curve represents the spline generated by the control points, while colored dotted lines indicate each control point’s contribution to embeddings, weighted by $b_{\eta,j}$. Colored triangles show the convex hulls formed by contributing control point groups.
• Figure 4: Validation graphs of Difformer and SCLM without KD. To ensure sufficient convergence, Difformer is trained for 500K iterations, and the resulting test scores are reported in Table \ref{['table:sacrebleu_results']}.
• Figure 5: Evaluation PPL ($\downarrow$) trends of MDLM and our SCLM during training.

Theorems & Definitions (13)

• Definition 3.1: Generic MLE Pipeline
• Lemma 3.2: The Optimal Solution of Maximum Likelihood Estimation
• Definition 4.1: Sentence Curve Prediction
• Lemma 4.2: MLE Objective in Sentence Curve Prediction (Simplified)
• Lemma 4.3
• Proposition 1.3: Continuous Relaxation of Words
• proof
• Lemma 1.4: The Optimal Solution of Maximum Likelihood Estimation
• proof
• Lemma 1.5: MLE Objective in Sentence Curve Prediction