VOGUE: A Multimodal Dataset for Conversational Recommendation in Fashion

TL;DR

Vogue addresses the challenge of evaluating multimodal conversational recommender systems in fashion by providing the first open dataset of real human–human dialogues grounded in shared visual catalogs, enriched with user profiles, item metadata, and explicit post-conversation ratings. The dataset enables analysis of item-level preference alignment and subjective satisfaction, and introduces stage-based dialogue annotations to study how recommendations evolve in context. Through human–AI benchmarking, the work demonstrates that while multimodal large language models can approximate aggregate alignment, they struggle with calibration and generalization across the full item catalog, underscoring fundamental gaps in modeling nuanced user preferences. Vogue thus offers a practical, rigorous benchmark to drive improvements in visual grounding, preference inference, and user-centric evaluation for conversational recommender systems in fashion.

Abstract

Multimodal conversational recommendation has emerged as a promising paradigm for delivering personalized experiences through natural dialogue enriched by visual and contextual grounding. Yet, current multimodal conversational recommendation datasets remain limited: existing resources either simulate conversations, omit user history, or fail to collect sufficiently detailed feedback, all of which constrain the types of research and evaluation they support. To address these gaps, we introduce VOGUE, a novel dataset of 60 humanhuman dialogues in realistic fashion shopping scenarios. Each dialogue is paired with a shared visual catalogue, item metadata, user fashion profiles and histories, and post-conversation ratings from both Seekers and Assistants. This design enables rigorous evaluation of conversational inference, including not only alignment between predicted and ground-truth preferences, but also calibration against full rating distributions and comparison with explicit and implicit user satisfaction signals. Our initial analyses of VOGUE reveal distinctive dynamics of visually grounded dialogue. For example, recommenders frequently suggest items simultaneously in feature-based groups, which creates distinct conversational phases bridged by Seeker critiques and refinements. Benchmarking multimodal large language models against human recommenders shows that while MLLMs approach human-level alignment in aggregate, they exhibit systematic distribution errors in reproducing human ratings and struggle to generalize preference inference beyond explicitly discussed items. These findings establish VOGUE as both a unique resource for studying multimodal conversational systems and as a challenge dataset beyond the current recommendation capabilities of existing top-tier multimodal foundation models such as GPT-4o-mini, GPT-5-mini, and Gemini-2.5-Flash.

Figures (12)

• Figure 1: Using Vogue, we evaluate a variety MLLMs on their ability to model user preferences against Human Assistants and two baselines: randomly sampled rating, and mode (i.e. most common) rating. Our results show that state-of-the-art MLLM preference generalization and understanding across the board substantially underperforms human agents.
• Figure 2: Study User Interfaces used in our experiment. Full-size images are available in the Appendix.
• Figure 3: Seeker & Assistant Item Mention Distribution by Conversation Proportion. We observe a bimodal distribution in the Assistant, corresponding to First Recommendation and Refinement stages respectively.
• Figure 4: Our proposed stage splits, illustrating Recommender--User and Seeker-Assistant initiative and major intent tags. Examples of dialogue are included, along with each role's motivation at each stage.
• Figure 5: Alignment metrics as a function of scenario order position. Scenario Order is the order in which individual trials were conducted for each Assistant-Seeker pair.