UniRec: Unified Multimodal Encoding for LLM-Based Recommendations

TL;DR

UniRec tackles heterogeneous multimodal signals in sequential recommendations by introducing modality-specific encoders, a triplet attribute representation, and a two-stage hierarchical Q-Former to preserve schema and history structure for next-item prediction $p(i_{T+1} \mid \mathcal{H}_u)$. It trains in two phases: first pretraining of the encoders with a frozen LLM using a reconstruction loss $\mathcal{L}_{\text{recon}}$ and a contrastive loss $\mathcal{L}_{\text{contrast}}$, then joint fine-tuning of the Q-Former and LoRA with the InfoNCE objective; the LLM is conditioned via soft prompts from the learned user representation. Empirical results on Beauty, Baby, and Yelp datasets show UniRec achieves state-of-the-art performance with up to 16% relative gains in MRR, and ablations validate the contributions of triplet schema, hierarchy, and query-based fusion. The approach generalizes LLM-based recommendation to fully heterogeneous signals and offers a scalable, schema-aware path toward more faithful reasoning over multimodal user–item histories.

Abstract

Large language models have recently shown promise for multimodal recommendation, particularly with text and image inputs. Yet real-world recommendation signals extend far beyond these modalities. To reflect this, we formalize recommendation features into four modalities: text, images, categorical features, and numerical attributes, and highlight the unique challenges this heterogeneity poses for LLMs in understanding multimodal information. In particular, these challenges arise not only across modalities but also within them, as attributes such as price, rating, and time may all be numeric yet carry distinct semantic meanings. Beyond this intra-modality ambiguity, another major challenge is the nested structure of recommendation signals, where user histories are sequences of items, each associated with multiple attributes. To address these challenges, we propose UniRec, a unified multimodal encoder for LLM-based recommendation. UniRec first employs modality-specific encoders to produce consistent embeddings across heterogeneous signals. It then adopts a triplet representation, comprising attribute name, type, and value, to separate schema from raw inputs and preserve semantic distinctions. Finally, a hierarchical Q-Former models the nested structure of user interactions while maintaining their layered organization. Across multiple real-world benchmarks, UniRec outperforms state-of-the-art multimodal and LLM-based recommenders by up to 15%, and extensive ablation studies further validate the contributions of each component.

Figures (3)

• Figure 1: UniRec Model Architecture: (a) Item-Level Q-Former: Raw item attributes across heterogeneous modalities (text, categorical, image, numerical) are processed by modality-specific encoders and triplet formation. These generate schema-aware attribute embeddings, which are then aggregated by the Item Q-Former to produce a fixed-length item representation ($\mathbf{z}_t$). (b) User-Level Q-Former: A user's chronological interaction history, consisting of learned item tokens ($\mathbf{z}_t$), multimodal review contexts ($\mathbf{c}_t$), and timestamp embeddings ($\mathbf{p}_t$), is processed by an Interaction Feature Assembly module. The resulting sequence of combined interaction embeddings is then distilled by the User Q-Former into a unified user representation ($\mathbf{U}$). The arrow passing the Learned Item Tokens from (a) to (b) explicitly models the nested structure of recommendation signals—where a user's history is a sequence of items, and each item is a collection of heterogeneous attributes. (c) LLM-Based Recommendation: The learned user representation and item representations are projected as soft prompts to condition the LLM for next-item prediction, ranking against a corpus of candidate item embeddings.
• Figure 2: Both schema- and hierarchy-aware components are crucial for UniRec’s performance. Results are shown on Beauty, Baby, and Yelp datasets (measured in MRR). Performance improves step by step as components are added: starting from the minimal configuration (w/o Both), introducing either triplet representation or user-level tokens yields clear gains, while combining both achieves the highest performance.
• Figure 3: Optimal token counts emerge for both item- and user-level Q-Formers. Left: item-level tokens. Right: user-level tokens. Each curve shows MRR on one dataset, and the red star ($\star$) marks the token count achieving the highest performance.