AR-Omni: A Unified Autoregressive Model for Any-to-Any Generation

TL;DR

AR-Omni introduces a unified autoregressive framework that enables any-to-any generation across text, image, and speech using a single Transformer decoder over a joint discrete vocabulary. By interleaving tokens from all modalities and applying task-aware loss weighting, a perceptual loss for image tokens, and a finite-state-like decoding strategy, it achieves real-time streaming for speech and diffusion-free image generation with competitive scores in image captioning, ASR, and TTS. The approach leverages text-based tokenization, acoustic speech tokens, and VQ-style image codes to create a cohesive autoregressive pipeline, supported by Omni-interleaved instruction data and carefully balanced pretraining data. While diffusion-free image generation lags diffusion-based methods in quality, AR-Omni demonstrates that a single AR backbone can practically unify perception and generation across modalities with real-time capabilities and strong cross-modal performance, pointing to future work on closing the image-generation gap without sacrificing AR simplicity.

Abstract

Real-world perception and interaction are inherently multimodal, encompassing not only language but also vision and speech, which motivates the development of "Omni" MLLMs that support both multimodal inputs and multimodal outputs. While a sequence of omni MLLMs has emerged, most existing systems still rely on additional expert components to achieve multimodal generation, limiting the simplicity of unified training and inference. Autoregressive (AR) modeling, with a single token stream, a single next-token objective, and a single decoder, is an elegant and scalable foundation in the text domain. Motivated by this, we present AR-Omni, a unified any-to-any model in the autoregressive paradigm without any expert decoders. AR-Omni supports autoregressive text and image generation, as well as streaming speech generation, all under a single Transformer decoder. We further address three practical issues in unified AR modeling: modality imbalance via task-aware loss reweighting, visual fidelity via a lightweight token-level perceptual alignment loss for image tokens, and stability-creativity trade-offs via a finite-state decoding mechanism. Empirically, AR-Omni achieves strong quality across three modalities while remaining real-time, achieving a 0.88 real-time factor for speech generation.

Figures (10)

• Figure 1: Overview of AR-Omni. Text, speech, and image inputs are tokenized and embedded into a shared space. A single autoregressive decoder operates over a joint vocabulary to generate a unified token stream. T denotes text, S denotes speech, and I denotes image.
• Figure 2: Stage 1 pretraining losses of AR-Omni. From left to right: weighted NTP loss, perceptual loss, and total loss. Curves are smoothed for readability and plotted over 1k--93k training steps.
• Figure 3: Loss curves of AR-Omni (Ours) and the simple NTP training objective. The naive objective exhibits a sharp loss jump, whereas AR-Omni maintains a smooth and stable loss throughout training.
• Figure 4: Case studies of AR-Omni: (a) multi-turn speech conversation (S$\rightarrow$S), (b) speech+image understanding with speech response (S+I$\rightarrow$S), and (c) speech-to-image generation (S$\rightarrow$I).
• Figure 5: Multi-turn interleaved conversation example.