PixRec: Leveraging Visual Context for Next-Item Prediction in Sequential Recommendation
Sayak Chakrabarty, Souradip Pal
TL;DR
PixRec addresses the limitations of text-only sequential recommenders in image-rich domains by introducing a vision–language framework for next-item prediction that augments textual attributes with product images. It uses a dual-tower vision–language backbone, a mixed objective combining next-item generation with cross-modal contrastive alignment, and a BM25-based retrieval layer, achieving large gains on image-augmented data. On an Amazon Reviews dataset with images, PixRec yields about a 3× improvement in top-rank accuracy and a 40% gain in top-10 accuracy over text-only baselines, demonstrating that visual cues help disambiguate items with near-identical text. The work outlines practical paths for scaling multi-modal training and managing inference latency, highlighting the potential for real-world e-commerce systems to leverage visual information in sequential recommendations.
Abstract
Large Language Models (LLMs) have recently shown strong potential for usage in sequential recommendation tasks through text-only models, which combine advanced prompt design, contrastive alignment, and fine-tuning on downstream domain-specific data. While effective, these approaches overlook the rich visual information present in many real-world recommendation scenarios, particularly in e-commerce. This paper proposes PixRec - a vision-language framework that incorporates both textual attributes and product images into the recommendation pipeline. Our architecture leverages a vision-language model backbone capable of jointly processing image-text sequences, maintaining a dual-tower structure and mixed training objective while aligning multi-modal feature projections for both item-item and user-item interactions. Using the Amazon Reviews dataset augmented with product images, our experiments demonstrate $3\times$ and 40% improvements in top-rank and top-10 rank accuracy over text-only recommenders respectively, indicating that visual features can help distinguish items with similar textual descriptions. Our work outlines future directions for scaling multi-modal recommenders training, enhancing visual-text feature fusion, and evaluating inference-time performance. This work takes a step toward building software systems utilizing visual information in sequential recommendation for real-world applications like e-commerce.