UniCMs: A Unified Consistency Model For Efficient Multimodal Generation and Understanding

TL;DR

<3-5 sentence high-level summary> UniCMs address the challenge of building a single consistency model capable of both efficient multimodal generation and understanding by defining a unified discrete denoising trajectory that combines image token diffusion with text token autoregressive parallel decoding. The framework uses a unified training objective with trajectory segmentation and regularization to encourage endpoint-consistent mappings across modalities, enabling fast-forward generation. Empirically, UniCMs achieve substantial speedups (e.g., ~1/8 sampling time for T2I) and competitive or superior accuracy on both T2I and MMU benchmarks, outperforming several strong baselines. The approach leverages Show-o-inspired trajectories and two-stage training on COCO and LLaVA data, highlighting strong practical impact for efficient multimodal AI systems.

Abstract

Consistency models (CMs) have shown promise in the efficient generation of both image and text. This raises the natural question of whether we can learn a unified CM for efficient multimodal generation (e.g., text-to-image) and understanding (e.g., image-to-text). Intuitively, such a model could be acquired by applying the consistency distillation (CD) to existing unified multimodal models. However, the key challenge is establishing a unified denoising perspective for both image and text generation, which is essential for establishing the consistency mapping. To tackle this, at the representation level, we advocate for discrete tokens for both modalities to best preserve language modeling capabilities. Critically, instead of defining the text denoising trajectory via recent discrete diffusion language modeling principles, we specify it using the parallel decoding trace of an autoregressive language model, benefiting from the latter's superior performance in general text generation tasks. The denoising trajectory of image tokens adheres to standard discrete diffusion. We train our unified consistency models (UniCMs) on these combined multimodal trajectories simultaneously with a unified objective. We introduce a trajectory segmentation strategy to further improve the training convergence. Empirically, in text-to-image generation, UniCMs outperform SD3 on GenEval, Image Reward, and CLIP Score metrics, while requiring only approximately ${1}/{8}$ of the sampling time. Meanwhile, in image-to-text generation, UniCMs surpass Show-o on the MMMU benchmark while being $1.5 \times$ faster at long-sequence generating speed. The code is available at https://github.com/zhijie-group/UniCMs.

Figures (9)

• Figure 1: 512 $\times$ 512 images generated by UniCMs. All images are generated by UniCMs in 4 sampling steps without reliance on classifier-free guidance ho2021classifier.
• Figure 2: Illustration of the unified denoising perspective of text and image generation. As shown, the trajectories both display a denoising pattern. The black line denotes the unified abstraction of the multimodal trajectory, and the red lines illustrate the objective of UniCMs---to map an arbitrary point on the sampling trajectory to the same endpoint for both text and image generation. Note that we omit the trajectory segmentation strategy in the training process for brevity.
• Figure 3: The text sampling trajectory of UniCMs in MMU cases. UniCMs realize acceleration by predicting multiple successive tokens in one iteration and correctly guessing the later tokens.
• Figure 4: Comparison between UniCMs, Show-o, and SD3 in T2I generation at the resolution of 512 $\times$ 512. Show-o is shown at 16 steps (using CFG), while UniCMs demonstrates performance at 8, 4, and 2 steps. SD3 results are included for comparison with UniCMs.
• Figure 5: Visualization of image inpainting by UniCMs on 256 resolution. From left to right are the 2, 4, and 8 steps sampling.