NoteLLM-2: Multimodal Large Representation Models for Recommendation

TL;DR

This work addresses multimodal item-to-item (I2I) recommendations by leveraging LLM-based representations. It identifies that naive end-to-end fine-tuning can bias models toward text and underutilize visual content, and introduces NoteLLM-2, which combines multimodal In-Context Learning (mICL) with a late fusion gate to preserve visual information in final representations. The approach enables end-to-end customization of LLMs and vision encoders for efficient multimodal representations, validated through extensive offline metrics and a week-long online A/B campaign showing increased user engagement. Overall, NoteLLM-2 provides a practical, scalable pathway to improve multimodal representations in recommendation systems and suggests future work extending to other modalities like video.

Abstract

Large Language Models (LLMs) have demonstrated exceptional proficiency in text understanding and embedding tasks. However, their potential in multimodal representation, particularly for item-to-item (I2I) recommendations, remains underexplored. While leveraging existing Multimodal Large Language Models (MLLMs) for such tasks is promising, challenges arise due to their delayed release compared to corresponding LLMs and the inefficiency in representation tasks. To address these issues, we propose an end-to-end fine-tuning method that customizes the integration of any existing LLMs and vision encoders for efficient multimodal representation. Preliminary experiments revealed that fine-tuned LLMs often neglect image content. To counteract this, we propose NoteLLM-2, a novel framework that enhances visual information. Specifically, we propose two approaches: first, a prompt-based method that segregates visual and textual content, employing a multimodal In-Context Learning strategy to balance focus across modalities; second, a late fusion technique that directly integrates visual information into the final representations. Extensive experiments, both online and offline, demonstrate the effectiveness of our approach. Code is available at https://github.com/Applied-Machine-Learning-Lab/NoteLLM.

Figures (6)

• Figure 1: Two strategies for using LLMs to enhance multimodal representations. The first strategy transfers existing pre-trained MLLMs into representation tasks. The second strategy fine-tunes the integration of LLMs and vision encoders end-to-end, requiring no alignment and offering better practicality and efficiency.
• Figure 2: The frameworks of MLRMs. (a) is the basic method for representations. (b) is our representation method, which contains two easy and effective mechanisms to enhance multimodal representation ability, mICL and late fusion.
• Figure 3: Relative sizes of $S_v$, $S_t$, and $S_o$ in different layers of different MLRMs. $S_v$ and $S_t$ are the mean significance of information flows from visual embeddings and textual embeddings to final compressed embeddings, respectively. $S_o$ is the mean significance of information flow among all embeddings except final compressed embeddings.
• Figure 4: Relative sizes of saliency scores in different layers of MLRMs. $S_v$ and $S_t$ are the mean significance of information flows from visual and textual embeddings to final compressed embeddings, respectively. $S_o$ is the mean significance of information flow among all embeddings except final compressed embeddings. $\hat{S_v}$, $\hat{S_t}$, and $\hat{S_o}$ are saliency scores enhanced by NoteLLM-2.
• Figure 5: The online serving pipeline includes offline and online phases. The offline phase generates embeddings with the MLRM trained with NoteLLM-2 and updates the ANN index with new notes. The online phase recalls notes from the index based on the user's interaction history.