HyperCLOVA X 8B Omni

TL;DR

HyperCLOVA X 8B Omni introduces an 8B-scale omnimodal decoder-only Transformer that unifies text, audio, and vision within a single interleaved token sequence. By combining discrete modality tokens with continuous encoders and diffusion-based decoders, the model achieves any-to-any multimodal understanding and generation, trained through a staged curriculum that starts with discrete tokens and progressively integrates continuous encoders and long-context capabilities. Empirical results across Korean and English benchmarks demonstrate competitive performance in text, vision-language, and audio tasks, including video understanding, with strong cross-lingual translation and TTS quality. The work offers a practical, open-weight omnimodal foundation that can scale with larger models and diverse deployment scenarios in both academia and industry.

Abstract

In this report, we present HyperCLOVA X 8B Omni, the first any-to-any omnimodal model in the HyperCLOVA X family that supports text, audio, and vision as both inputs and outputs. By consolidating multimodal understanding and generation into a single model rather than separate modality-specific pipelines, HyperCLOVA X 8B Omni serves as an 8B-scale omni-pathfinding point toward practical any-to-any omni assistants. At a high level, the model unifies modalities through a shared next-token prediction interface over an interleaved multimodal sequence, while vision and audio encoders inject continuous embeddings for fine-grained understanding and grounding. Empirical evaluations demonstrate competitive performance against comparably sized models across diverse input-output combinations spanning text, audio, and vision, in both Korean and English. We anticipate that the open-weight release of HyperCLOVA X 8B Omni will support a wide range of research and deployment scenarios.

Figures (10)

• Figure 1: Comparison of multimodal capabilities across text, vision, and audio for both generation and understanding tasks. The results highlight the unified end-to-end design of HyperCLOVA X 8B Omni, which supports any-to-any multimodal understanding and generation within a single model.
• Figure 2: Overall architecture of HyperCLOVA X 8B Omni. Text, vision, and audio inputs are encoded into continuous embeddings and discrete tokens via modality-specific encoders and tokenizers, which are interleaved and jointly processed by a single decoder-only Transformer backbone. Modality-specific decoders reconstruct visual and auditory outputs from the shared sequence representations, enabling end-to-end any-to-any multimodal generation.
• Figure 3: (Left) Reconstruction test of TA-Tok han_vision_2025 using its accompanying decoder. The reconstruction is imperfect due to unavoidable information loss from semantic abstraction and quantization (see the eye and the feather pattern of the bird, and the tonal difference in both cases). A non-square image shown at the bottom is "resized to square $\rightarrow$ tokenized $\rightarrow$ decoded as a square image $\rightarrow$ resized back to the original aspect ratio." We observe that the distortion level is not critical and presume that it could be compensated for by training a new decoder reflecting this process. (Right) Convergence of validation loss for the classic attention-based architecture (green) and our channel-concatenation-based architecture (blue).
• Figure 4: Overview of the training process. The model is first trained using discrete modality tokens for text, vision, and audio, establishing a unified symbolic token interface across modalities. Continuous vision and audio encoders are then integrated and jointly trained alongside the discrete tokens, enabling richer multimodal perception within the same Transformer backbone.
• Figure 5: Distribution of the post-training datasets across four training stages. Stage 1 focuses on foundational conversational alignment; Stage 2 expands to task-oriented multimodal instructions; Stage 3 introduces temporal and long-context understanding; and Stage 4 refines user-intent reasoning through integrated reasoning paths.