Meta-DiffuB: A Contextualized Sequence-to-Sequence Text Diffusion Model with Meta-Exploration

TL;DR

The Meta-DiffuB framework is proposed - a novel scheduler-exploiter S2S-Diffusion paradigm designed to overcome the limitations of existing S2S-Diffusion models and achieves state-of-the-art performance compared to previous S2S-Diffusion models and fine-tuned pre-trained language models (PLMs) across four Seq2Seq benchmark datasets.

Abstract

The diffusion model, a new generative modeling paradigm, has achieved significant success in generating images, audio, video, and text. It has been adapted for sequence-to-sequence text generation (Seq2Seq) through DiffuSeq, termed S2S Diffusion. Existing S2S-Diffusion models predominantly rely on fixed or hand-crafted rules to schedule noise during the diffusion and denoising processes. However, these models are limited by non-contextualized noise, which fails to fully consider the characteristics of Seq2Seq tasks. In this paper, we propose the Meta-DiffuB framework - a novel scheduler-exploiter S2S-Diffusion paradigm designed to overcome the limitations of existing S2S-Diffusion models. We employ Meta-Exploration to train an additional scheduler model dedicated to scheduling contextualized noise for each sentence. Our exploiter model, an S2S-Diffusion model, leverages the noise scheduled by our scheduler model for updating and generation. Meta-DiffuB achieves state-of-the-art performance compared to previous S2S-Diffusion models and fine-tuned pre-trained language models (PLMs) across four Seq2Seq benchmark datasets. We further investigate and visualize the impact of Meta-DiffuB's noise scheduling on the generation of sentences with varying difficulties. Additionally, our scheduler model can function as a "plug-and-play" model to enhance DiffuSeq without the need for fine-tuning during the inference stage.

Figures (4)

• Figure 1: Comparison between S2S-Diffusion model (i.e., DiffuSeq 2022diffusionlm2022) and the proposed Meta-Diffu$B$. The shades of color represent different amounts of noise being imposed. Different from prior works that use a fixed noise, we introduce a novel scheduler-exploiter framework, Meta-Diffu$B$, which achieves trainable noise scheduling inspired by Meta Exploration. Our scheduler model schedules contextualized noise, enhancing the training and generation of the S2S-Diffusion model, resulting in state-of-the-art (SOTA) performance compared to previous S2S-Diffusion models, as detailed in Section \ref{['sec: Experimental']}.
• Figure 2: Increase in BLEU score with varying candidate sizes $|S|$ on the QQP and WA datasets.
• Figure 3: Visualization of noise scheduling for each S2S-Diffusion model on the QQP and WA datasets. $\overline{\beta}_t$ represents the average noise imposed on sentences at diffusion step $t$. Unlike other models, which impose the same noise on all sentences, our Meta-Diffu$B$ ($D_\theta$ = DiffuSeq) varies the noise levels.
• Figure 4: Adaptive noise scheduling for each S2S-Diffusion model on the QQP and WA datasets. $\Sigma \boldsymbol{\beta}$ represents the total amount of noise imposed in each training epoch.