Benchmarking Multimodal Large Language Models for Missing Modality Completion in Product Catalogues

TL;DR

This work tackles missing-modality completion in e-commerce product catalogs by introducing MMPCBench, a dual-benchmark suite with Content Quality (CQBench) and Recommendation (RecBench) components to evaluate how well Multimodal Large Language Models can generate missing images or text. It systematically evaluates six MLLMs from the Qwen2.5-VL and Gemma-3 families across nine product categories for both image-to-text and text-to-image completion, and investigates Group Relative Policy Optimization (GRPO) to improve task alignment. Key findings show that while MLLMs capture high-level semantics, they struggle with fine-grained word- and pixel-level fidelity, and model size effects are non-monotonic; GRPO improves image-to-text but provides limited gains for text-to-image. The paper provides an open, reproducible benchmark and offers insights into the practical viability and limitations of missing-modality generation in real-world catalogs, with GRPO identified as a promising direction for I→T alignment but not yet for T→I.

Abstract

Missing-modality information on e-commerce platforms, such as absent product images or textual descriptions, often arises from annotation errors or incomplete metadata, impairing both product presentation and downstream applications such as recommendation systems. Motivated by the multimodal generative capabilities of recent Multimodal Large Language Models (MLLMs), this work investigates a fundamental yet underexplored question: can MLLMs generate missing modalities for products in e-commerce scenarios? We propose the Missing Modality Product Completion Benchmark (MMPCBench), which consists of two sub-benchmarks: a Content Quality Completion Benchmark and a Recommendation Benchmark. We further evaluate six state-of-the-art MLLMs from the Qwen2.5-VL and Gemma-3 model families across nine real-world e-commerce categories, focusing on image-to-text and text-to-image completion tasks. Experimental results show that while MLLMs can capture high-level semantics, they struggle with fine-grained word-level and pixel- or patch-level alignment. In addition, performance varies substantially across product categories and model scales, and we observe no trivial correlation between model size and performance, in contrast to trends commonly reported in mainstream benchmarks. We also explore Group Relative Policy Optimization (GRPO) to better align MLLMs with this task. GRPO improves image-to-text completion but does not yield gains for text-to-image completion. Overall, these findings expose the limitations of current MLLMs in real-world cross-modal generation and represent an early step toward more effective missing-modality product completion.

Figures (5)

• Figure 1: Missing-Modality Completion for E-Commerce Products: Given only one modality (either image or text), the task is to generate the missing counterpart. Missing modalities lead to incomplete product presentation, which can negatively impact downstream tasks such as representation learning and recommendation.
• Figure 2: Overview of MMPCBench. MMPCBench is a benchmark designed to evaluate MLLMs on the task of completing missing product modalities—either text or image—across diverse product categories. EMB denotes the embedding models used for similarity-based evaluation, while RM refers to the recommender models employed for downstream recommendation tasks. The generated modalities are evaluated using these two components of the benchmark. Notably, completing missing images requires a diffusion model to visualize the image prompts generated by the MLLMs.
• Figure 3: Radar Charts for Both T$\rightarrow$I and I$\rightarrow$T Tasks on Content Quality Benchmark. Each metric is normalized to a [0, 1] range and then summed, resulting in a maximum score of 4 for I$\rightarrow$T and 5 for T$\rightarrow$I tasks.
• Figure 4: Heatmap Illustrating the Differences Among MLLM Completion Methods Across Various Recommendation Categories. Colors closer to blue indicate greater performance degradation compared to using the ground-truth multimodal inputs.
• Figure 5: Reward Curves of different tasks.