MoE-DiffuSeq: Enhancing Long-Document Diffusion Models with Sparse Attention and Mixture of Experts
Alexandros Christoforos, Chadbourne Davis
TL;DR
Long-document diffusion-based generation faces prohibitive compute and memory costs. The paper introduces MoE-DiffuSeq, which fuses Mixture of Experts with sparse attention within the DiffuSeq framework and adds a soft absorbing state to accelerate diffusion-based generation. Across multiple long-context benchmarks, the model delivers substantial gains in efficiency, sampling speed, and semantic quality, outperforming DiffuSeq and Longformer baselines. Ablation studies confirm the critical role of sparse attention and MoE in achieving robust long-form text generation at scale.
Abstract
We present MoE-DiffuSeq, a mixture of experts based framework for enhancing diffusion models in long document generation. Existing diffusion based text generation models, such as DiffuSeq, suffer from high computational cost and memory overhead when applied to extended sequences. To address these challenges, MoE-DiffuSeq integrates sparse attention with a mixture of experts architecture, enabling efficient and scalable long sequence modeling. Our approach introduces a customized sparse attention mechanism designed to reduce computational complexity while preserving text quality and coherence. In addition, we incorporate a soft absorbing state within the diffusion process to accelerate sequence reconstruction and improve generation precision. Extensive experiments demonstrate that MoE-DiffuSeq significantly improves training efficiency and sampling speed compared to existing diffusion models. These advantages are particularly effective for long document scenarios, including scientific article generation, code repository modeling, and long form dialogue generation. Benchmark results further show that MoE-DiffuSeq improves efficiency, speed, accuracy, and expressiveness, advancing the practical applicability of diffusion models for high quality long form text generation.