MOON: Generative MLLM-based Multimodal Representation Learning for E-commerce Product Understanding

TL;DR

MOON introduces a generative MLLM-based framework for general e-commerce product representation learning, addressing many-to-one image-text alignment and background noise via core-product detection, a guided Mixture-of-Experts, and spatial-temporal negative sampling. The model is trained with user-behavior-based contrastive learning and evaluated on the new MBE benchmark, achieving strong zero-shot performance across retrieval, classification, and attribute prediction. A large-scale real-world dataset with hierarchical categories and attributes supports realistic evaluation, and attention visualizations demonstrate cross-modal alignment and interpretability. Overall, the work broadens the applicability of generative MLLMs in e-commerce and provides a standardized benchmark to accelerate future research.

Abstract

With the rapid advancement of e-commerce, exploring general representations rather than task-specific ones has attracted increasing research attention. For product understanding, although existing discriminative dual-flow architectures drive progress in this field, they inherently struggle to model the many-to-one alignment between multiple images and texts of products. Therefore, we argue that generative Multimodal Large Language Models (MLLMs) hold significant potential for improving product representation learning. Nevertheless, achieving this goal still remains non-trivial due to several key challenges: the lack of multimodal and aspect-aware modeling modules in typical LLMs; the common presence of background noise in product images; and the absence of a standard benchmark for evaluation. To address these issues, we propose the first generative MLLM-based model named MOON for product representation learning. Our method (1) employs a guided Mixture-of-Experts (MoE) module for targeted modeling of multimodal and aspect-specific product content; (2) effectively detects core semantic regions in product images to mitigate the distraction and interference caused by background noise; and (3) introduces the specialized negative sampling strategy to increase the difficulty and diversity of negative samples. In addition, we release a large-scale multimodal benchmark MBE for various product understanding tasks. Experimentally, our model demonstrates competitive zero-shot performance on both our benchmark and the public dataset, showcasing strong generalization across various downstream tasks, including cross-modal retrieval, product classification, and attribute prediction. Furthermore, the case study and visualization illustrate the effectiveness of MOON for product understanding.

Figures (9)

• Figure 1: Overall results on all the downstream tasks.
• Figure 2: Comparison between the dual-flow and MLLM architectures. (a) The dual-flow paradigm is inherently limited to encode one-to-one image-text pairs and cannot directly capture many-to-one relationships. (b) The MLLM-based idea is naturally suited to model the richer visual content from multiple SKU images.
• Figure 3: Illustration of the noisy background and the core product. (a) Besides the product itself, images often include non-sale objects and background noise. (b) The core semantics of the products for eliminating the distractions of MLLMs.
• Figure 4: The architecture of our MOON. (a) We leverage real-world purchase behaviors as supervision to effectively capture latent correlations between related items. Moreover, we employ the spatial and temporal negative sampling, adding hard negative samples and fully expanding the negative pool, to learn more robust and discriminative representations. (b) Beyond the dual-encoder paradigm, we propose the first generative MLLM-based method for product understanding. A product detector is used to crop the core semantic region, which is input to the MLLM alongside the original image. Finally, the mean pooling of the LLM's last hidden states serves as the product representation.
• Figure 5: The guided MoE module. We replace the FFN layer with a guided MoE module, which explicitly designate two specialized experts to handle the category and attributes within the text input.

Theorems & Definitions (3)

• definition 1
• definition 2
• definition 3