SAIL-Embedding Technical Report: Omni-modal Embedding Foundation Model

TL;DR

SAIL-Embedding tackles the challenge of omni-modal retrieval and recommendation by integrating vision, text, and audio into a unified embedding via a central LLM fusion backbone. The model employs dynamic hard negative mining, adaptive multi-source data balancing, content-aware progressive training, and collaboration-aware recommendation enhancement to achieve state-of-the-art results across item-to-item and query-to-item tasks, while delivering measurable online lifetime gains in real-world Douyin scenarios. Comprehensive ablations and online studies validate the necessity and effectiveness of each module, including sequence-to-item and ID-to-item distillation for enhanced user-interest modeling. The work demonstrates practical impact for industrial multimodal systems, offering scalable training strategies and flexible pattern-matching frameworks that generalize to diverse downstream tasks such as video understanding and cross-domain knowledge transfer.

Abstract

Multimodal embedding models aim to yield informative unified representations that empower diverse cross-modal tasks. Despite promising developments in the evolution from CLIP-based dual-tower architectures to large vision-language models, prior works still face unavoidable challenges in real-world applications and business scenarios, such as the limited modality support, unstable training mechanisms, and industrial domain gaps. In this work, we introduce SAIL-Embedding, an omni-modal embedding foundation model that addresses these issues through tailored training strategies and architectural design. In the optimization procedure, we propose a multi-stage training scheme to boost the multifaceted effectiveness of representation learning. Specifically, the content-aware progressive training aims to enhance the model's adaptability to diverse downstream tasks and master enriched cross-modal proficiency. The collaboration-aware recommendation enhancement training further adapts multimodal representations for recommendation scenarios by distilling knowledge from sequence-to-item and ID-to-item embeddings while mining user historical interests. Concurrently, we develop the stochastic specialization and dataset-driven pattern matching to strengthen model training flexibility and generalizability. Experimental results show that SAIL-Embedding achieves SOTA performance compared to other methods in different retrieval tasks. In online experiments across various real-world scenarios integrated with our model, we observe a significant increase in Lifetime (LT), which is a crucial indicator for the recommendation experience. For instance, the model delivers the 7-day LT gain of +0.5% in the Douyin-Selected scenario. For the Douyin feed rank model, the match features produced by SAIL-Embedding yield a +0.1% AUC gain.

Figures (10)

• Figure 1: SAIL-Embedding Capability Overview. We present SAIL-Embedding, an omni-modal embedding model adapted from vision-language-audio models. SAIL-Embedding is capable of following instructions and performing various omni-modal embedding tasks, such as Motivation/Tag-CLS (Classification), Search-q2i (Query-to-Item Retrieval), and Copair/Live-i2i (Item-to-Item Retrieval).
• Figure 2: Embedding Architecture Comparison. (a) CLIP-like dual-tower embedding model architecture. (b) LLM/MLLM-based embedding model architecture. (c) SAIL-Embedding model architecture. In contrast, our model accommodates arbitrary modality inputs and can handle diverse downstream tasks. I, V, T, and O represent image, video, text, and omni-modality, respectively.
• Figure 3: Illustration of Adaptive Multi-Source Data Balancing. For both training and validation datasets, we compute embeddings for different modalities and cluster them. We then construct a similarity matrix to represent the distance between training and validation clusters, which is further aggregated into a single similarity score. Based on these similarity values, each training dataset is assigned a weight using a Softmax function.
• Figure 4: Overview of Our SAIL-Embedding Model. For each sample, we extract relevant textual information (e.g., titles, OCR text) and combine it with instructions to obtain textual tokens. Visual and audio information are encoded using dedicated encoders, and the resulting multimodal tokens are concatenated and passed through a fusion module to obtain a unified multimodal embedding. The system is trained by contrasting embeddings from query and target samples.
• Figure 5: Illustration of instructions for different tasks. We design task-specific instructions by explicitly defining the task and its objective. Audio, image, and text tokens are then provided jointly to obtain the final results. For query and target tokens, modality-specific adaptations are applied to accommodate their respective modality combinations.