LEMUR: Large scale End-to-end MUltimodal Recommendation

TL;DR

LEMUR presents the first fully end-to-end multimodal recommender trained directly from raw data at industrial scale. By integrating multimodal encoding with ranking, and introducing a memory bank for efficient construction of long multimodal histories alongside a session-masked contrastive loss (SQDC), it achieves tighter alignment with downstream objectives and real-time adaptability. Empirically, LEMUR delivers substantial offline gains and real-world online improvements (e.g., 0.843% reduction in query change rate and 0.81% QAUC uplift) on Douyin platforms, outperforming two-stage baselines. The work demonstrates the practicality of end-to-end multimodal optimization for large-scale recommender systems and outlines practical considerations, such as staleness and coverage in memory banks, for future refinement.

Abstract

Traditional ID-based recommender systems often struggle with cold-start and generalization challenges. Multimodal recommendation systems, which leverage textual and visual data, offer a promising solution to mitigate these issues. However, existing industrial approaches typically adopt a two-stage training paradigm: first pretraining a multimodal model, then applying its frozen representations to train the recommendation model. This decoupled framework suffers from misalignment between multimodal learning and recommendation objectives, as well as an inability to adapt dynamically to new data. To address these limitations, we propose LEMUR, the first large-scale multimodal recommender system trained end-to-end from raw data. By jointly optimizing both the multimodal and recommendation components, LEMUR ensures tighter alignment with downstream objectives while enabling real-time parameter updates. Constructing multimodal sequential representations from user history often entails prohibitively high computational costs. To alleviate this bottleneck, we propose a novel memory bank mechanism that incrementally accumulates historical multimodal representations throughout the training process. After one month of deployment in Douyin Search, LEMUR has led to a 0.843% reduction in query change rate decay and a 0.81% improvement in QAUC. Additionally, LEMUR has shown significant gains across key offline metrics for Douyin Advertisement. Our results validate the superiority of end-to-end multimodal recommendation in real-world industrial scenarios.

Figures (3)

• Figure 1: The Overall Framework of LEMUR.
• Figure 2: The Structure of Decoder
• Figure 3: Staleness and coverage in memory bank.