LaViC: Adapting Large Vision-Language Models to Visually-Aware Conversational Recommendation

TL;DR

LaViC tackles visually-informed conversational recommendation by addressing token explosion in multimodal LLMs. It introduces a two-stage approach: visual knowledge self-distillation compresses each item image into a small set of embeddings that preserve key appearance details, and recommendation prompt tuning trains a frozen LM to select the correct item from compact multimodal representations. The authors validate the method on the Reddit-Amazon dataset across beauty, fashion, and home, showing substantial improvements over text-only and open-source vision-language baselines, and competitive performance with proprietary models using a 7B backbone. This work demonstrates that explicit visual data integration with compact representations yields significant gains in visually-driven domains while maintaining tractable compute, enabling practical deployment in conversational recommender systems. The Reddit-Amazon dataset and code availability further provide a valuable resource for evaluating visually-aware CRS in real-world contexts.

Abstract

Conversational recommender systems engage users in dialogues to refine their needs and provide more personalized suggestions. Although textual information suffices for many domains, visually driven categories such as fashion or home decor potentially require detailed visual information related to color, style, or design. To address this challenge, we propose LaViC (Large Vision-Language Conversational Recommendation Framework), a novel approach that integrates compact image representations into dialogue-based recommendation systems. LaViC leverages a large vision-language model in a two-stage process: (1) visual knowledge self-distillation, which condenses product images from hundreds of tokens into a small set of visual tokens in a self-distillation manner, significantly reducing computational overhead, and (2) recommendation prompt tuning, which enables the model to incorporate both dialogue context and distilled visual tokens, providing a unified mechanism for capturing textual and visual features. To support rigorous evaluation of visually-aware conversational recommendation, we construct a new dataset by aligning Reddit conversations with Amazon product listings across multiple visually oriented categories (e.g., fashion, beauty, and home). This dataset covers realistic user queries and product appearances in domains where visual details are crucial. Extensive experiments demonstrate that LaViC significantly outperforms text-only conversational recommendation methods and open-source vision-language baselines. Moreover, LaViC achieves competitive or superior accuracy compared to prominent proprietary baselines (e.g., GPT-3.5-turbo, GPT-4o-mini, and GPT-4o), demonstrating the necessity of explicitly using visual data for capturing product attributes and showing the effectiveness of our vision-language integration. Our code and dataset are available at https://github.com/jeon185/LaViC.

Figures (5)

• Figure 1: The Reddit-Amazon dataset contains conversations between a seeker and a recommender. The seeker requests an item recommendation via text, focusing primarily on visual preferences. The item recommended to the seeker by the recommender is linked to detailed item information such as its title and images.
• Figure 2: Illustration of visual knowledge self-distillation. Generation process (left): The vision tower and projector encode each sub-image into hundreds of patch embeddings (577 for each sub-image in LLaVA-v1.6), which are passed to the large language model (LLM) alongside a textual prompt. The LLM then produces a detailed product description focusing on visual features. Distillation process (right): We freeze the LLM and train only the vision tower and projector (via LoRA) to condense each sub-image into a single [CLS]-positioned embedding, yet still generate the same descriptive text. This reduces the token count from thousands to a handful, minimizing computational overhead while retaining essential visual information.
• Figure 3: The validation perplexity reaches a plateau after 1--2 epochs.
• Figure 4: Illustration of recommendation prompt tuning. We integrate the compressed embeddings with conversation and item IDs/titles for recommendation. We then train the large language model (LLM) using LoRA while fixing the parameters of vision tower and projector.
• Figure 5: Two cases comparing LaViC with LLaVA-v1.6. Both cases are selected from fashion domain. The ✓ indicates a correct recommendation, while the ✗ denotes an incorrect one. (a) LaViC identifies a more casual "G.H. Bass & Co. Men’s Buckingham Oxford", whereas LLaVA-v1.6 either suggests a more formal shoe under the same brand or a different brand entirely. (b) LaViC recommends a backpack that meets the user’s specific strap requirements (two separate shoulder straps and a cross-chest strap), which LLaVA-v1.6 (w/ title & image) fails to satisfy, even with image input.