RAG-VisualRec: An Open Resource for Vision- and Text-Enhanced Retrieval-Augmented Generation in Recommendation

TL;DR

RAG-VisualRec is introduced, an open resource and reproducible pipeline that combines LLM-generated item-side plot descriptions and trailer-derived visual embeddings, supporting both retrieval-augmented generation (RAG) and collaborative-filtering style workflows, and a complementary analysis that increases transparency and reproducibility is provided.

Abstract

This paper addresses the challenge of building multimodal recommender systems for the movie domain, where sparse item metadata (e.g., title and genres) can limit retrieval quality and downstream recommendations. We introduce RAG-VisualRec, an open resource and reproducible pipeline that combines (i) LLM-generated item-side plot descriptions and (ii) trailer-derived visual (and optional audio) embeddings, supporting both retrieval-augmented generation (RAG) and collaborative-filtering style workflows. Our pipeline augments sparse metadata into richer textual signals and integrates modalities via configurable fusion strategies (e.g., PCA and CCA) before retrieval and optional LLM-based re-ranking. Beyond providing the resource, we provide a complementary analysis that increases transparency and reproducibility. In particular, we introduce LLMGenQC, a critic-based quality-control module (LLM-as-judge) that audits synthetic synopses for semantic alignment with metadata, consistency, safety, and basic sanity checks, releasing critic scores and pass/fail labels alongside the generated artifacts. We report ablation studies that quantify the impact of key design choices, including retrieval depth, fusion strategy, and user-embedding construction. Across experiments, CCA-based fusion consistently improves recall over unimodal baselines, while LLM-based re-ranking typically improves nDCG by refining top-K selection from the retrieved candidate pool, especially when textual evidence is limited. By releasing RAG-VisualRec, we enable further research on multimodal RAG recommenders, quality auditing of LLM-generated side information, and long-tail oriented evaluation protocols. All code, data, and detailed documentation are publicly available at: https://github.com/RecSys-lab/RAG-VisualRec.

Figures (10)

• Figure 1: The overall pipeline of the proposed multimodal RAG framework.
• Figure 2: Overall performance overview.
• Figure 3: Visual-RAG: Modular architecture supporting data ingestion, multimodal embedding, fusion, LLM-based data augmentation, critic-based quality auditing of synthetic texts (LLMGenQC ), retrieval, LLM re-ranking, and robust evaluation.
• Figure 4: t-SNE projection of item and user embeddings using the Sentence Transformer (ST) LLM backbone. Multimodal embeddings are obtained via CCA.
• Figure 5: Radar plot illustrating the performance of the recommendation stage across both accuracy and beyond-accuracy metrics for various LLMs.