D5P4: Partition Determinantal Point Process for Diversity in Parallel Discrete Diffusion Decoding

Abstract

Discrete diffusion models are promising alternatives to autoregressive approaches for text generation, yet their decoding methods remain under-studied. Standard decoding methods for autoregressive models, such as beam search, do not directly apply to iterative denoising, and existing diffusion decoding techniques provide limited control over in-batch diversity. To bridge this gap, we introduce a generalized beam-search framework for discrete diffusion that generates candidates in parallel and supports modular beam-selection objectives. As a diversity-focused instantiation, we propose D5P4, which formulates the selection step as MAP inference over a Determinantal Point Process. Leveraging a scalable greedy solver, D5P4 maintains multi-GPU compatibility and enables an explicit trade-off between model probability and target diversity with near-zero compute overhead. Experiments on free-form generation and question answering demonstrate that D5P4 improves diversity over strong baselines while maintaining competitive generation quality.

Figures (10)

• Figure 1: Overview of the discrete diffusion beam search algorithm. Partially denoised sequences at time $t$ are fed to the model $p_\theta$. The model produces logits and embeddings that are then used in the Joint Scoring & Groupwise Selection block. The logits of the selected sequences for each group are expanded through independent applications of the projection operator $\Pi$, resulting in the sequence at time $s$.
• Figure 2: Pareto front comparison of diversity-encouraging methods in open-ended generation. Lower perplexity indicates higher quality, and lower cosine similarity indicates higher diversity. The single point for baseline corresponds to an independent sampling baseline. CAT is the categorical temperature modulation. DivBS is the transversal MMR search. Our methods approximate the MAP of an additive and multiplicative kernel. D5P4 consistently achieves better diversity-quality trade-offs.
• Figure 3: Effect of diversity control parameter $\beta$ on distribution fidelity (MAUVE and MAUVE*), using D5P4$+$.
• Figure 4: Mitigation of CFG diversity collapse. Increasing CFG strength typically reduces diversity (higher EAD). D5P4 counteracts this collapse, maintaining higher diversity across CFG values.
• Figure 5: Correlation between MDLM-estimated log-likelihood and GPT-2 log-likelihood across samples.