NExT-OMNI: Towards Any-to-Any Omnimodal Foundation Models with Discrete Flow Matching

TL;DR

NExT-OMNI introduces a fully open-source omnimodal foundation model based on discrete flow matching to enable any-to-any cross-modal understanding and generation across text, image, video, and audio. The method unifies modalities with warmup reconstruction and semantic-alignment losses, followed by a discrete flow matching objective that leverages metric-based probability paths and a coarse-to-fine training strategy. Key contributions include a unified architecture initialized from LLMs, reconstruction-enhanced unified representations, and comprehensive evaluations showing competitive or superior performance with lower latency than autoregressive counterparts. The work also provides training/data protocols and open-source code/checkpoints to accelerate research in unified multimodal modeling. Overall, it presents a promising shift from autoregressive fusion to a discrete-flow paradigm for scalable, multi-turn omnimodal intelligence.

Abstract

Next-generation multimodal foundation models capable of any-to-any cross-modal generation and multi-turn interaction will serve as core components of artificial general intelligence systems, playing a pivotal role in human-machine interaction. However, most existing multimodal models remain constrained by autoregressive architectures, whose inherent limitations prevent a balanced integration of understanding and generation capabilities. Although hybrid and decoupling strategies have been explored to address these tasks within unified frameworks separately, their redundant, non-integrated designs limit their applicability to broader scenarios, such as cross-modal retrieval. In this work, we introduce NExT-OMNI, an open-source omnimodal foundation model that achieves unified modeling through discrete flow paradigms. By leveraging metric-induced probability paths and kinetic optimal velocities, NExT-OMNI natively supports any-to-any understanding and generation with enhanced response efficiency, while enabling broader application scenarios through concise unified representations rather than task-decoupled designs. Trained on large-scale interleaved text, image, video, and audio data, NExT-OMNI delivers competitive performance on multimodal generation and understanding benchmarks, while outperforming prior unified models in multi-turn multimodal interaction and cross-modal retrieval, highlighting its architectural advantages as a next-generation multimodal foundation model. To advance further research, we release training details, data protocols, and open-source both the code and model checkpoints.

Figures (22)

• Figure 1: Overview of the NExT-OMNI framework. NExT-OMNI is a unified framework for omnimodal tasks, offering strong understanding, generation, and retrieval capabilities. The framework enables any-to-any operations across modalities for both generation and understanding tasks, while achieving accelerated processing. It also supports cross-modal retrieval by leveraging rich multimodal representations aggregated within its architecture.
• Figure 2: Pipeline of the NExT-OMNI framework. NExT-OMNI employs a DFM paradigm for unified omnimodal training, with multimodal self-attention at every layer to deeply fuse information across modalities. Unlike prior methods using multiple encoders or mixture-of-experts, it trains a single encoder simultaneously for understanding and generation, producing unified representations that enable any-to-any multimodal tasks with a streamlined architecture and strong generalization across diverse scenarios.
• Figure 3: Illustrations of dynamic generation strategy (left) and vanilla adaptive cache design (right). During training, responses are padded to multiples of the block size, allowing the model to extend preset response lengths in block-size increments during inference. This improves performance on understanding tasks that require dynamic-length generation. The vanilla adaptive cache caches instruction features and selectively caches response features based on feature cosine similarity, substantially accelerating inference and decoding. Combined with parallel decoding, this simple yet efficient caching design enables NExT-OMNI to generate responses faster than AR-based models.
• Figure 4: The Pipeline of vision encoder (left) and audio encoder (right) warmup training. During the modal encoder warmup training phase, self-supervised reconstruction is performed through additional quantizers and modal decoders, while semantic alignment at different granularities is achieved through text encoders or text decoders.
• Figure 5: Qualitative comparison results on audio reconstruction in a max duration of 15s.