End-to-end training of Multimodal Model and ranking Model

TL;DR

EM3 introduces End-to-end training of Multimodal Model and ranking Model for industrial recommender systems. It presents Fusion-Q-Former to fuse visual and textual modalities into fixed-length embeddings, employs LoRA to extend sequence modeling efficiently, and adds Content-ID-Contrastive learning to align content and ID embeddings. The approach yields consistent offline AUC gains and significant online improvements in GMV, CTR, and revenue across two production scenarios, with ablations confirming the contributions of each component. Public dataset experiments further demonstrate state-of-the-art performance and generalizability, while a reusable codebase supports reproducibility. The work has practical impact by enabling task-oriented content features through integrated multimodal representations in production ranking models, with potential extensions to additional modalities and end-to-end multimodal large language models.

Abstract

Traditional recommender systems heavily rely on ID features, which often encounter challenges related to cold-start and generalization. Modeling pre-extracted content features can mitigate these issues, but is still a suboptimal solution due to the discrepancies between training tasks and model parameters. End-to-end training presents a promising solution for these problems, yet most of the existing works mainly focus on retrieval models, leaving the multimodal techniques under-utilized. In this paper, we propose an industrial multimodal recommendation framework named EM3: End-to-end training of Multimodal Model and ranking Model, which sufficiently utilizes multimodal information and allows personalized ranking tasks to directly train the core modules in the multimodal model to obtain more task-oriented content features, without overburdening resource consumption. First, we propose Fusion-Q-Former, which consists of transformers and a set of trainable queries, to fuse different modalities and generate fixed-length and robust multimodal embeddings. Second, in our sequential modeling for user content interest, we utilize Low-Rank Adaptation technique to alleviate the conflict between huge resource consumption and long sequence length. Third, we propose a novel Content-ID-Contrastive learning task to complement the advantages of content and ID by aligning them with each other, obtaining more task-oriented content embeddings and more generalized ID embeddings. In experiments, we implement EM3 on different ranking models in two scenario, achieving significant improvements in both offline evaluation and online A/B test, verifying the generalizability of our method. Ablation studies and visualization are also performed. Furthermore, we also conduct experiments on two public datasets to show that our proposed method outperforms the state-of-the-art methods.

Figures (4)

• Figure 1: The overall framework of EM3. (a) Multimodal Model: aims to fuse modalities and shares parameters across different items. (b) CIC: a self-supervised task to align content and ID by maximizing the cosine similarities of the pairs on the diagonal.
• Figure 2: The details of LoRA module. The right $W ^ {d \times d}$ is frozen, while the left $W_a ^{d\times r}$ and $W_b^{r\times d}$ are trainable.
• Figure 3: The feature system of EM3.
• Figure 4: Visualization of the impacts on embeddings.