Transformer-empowered Multi-modal Item Embedding for Enhanced Image Search in E-Commerce

TL;DR

This work tackles the limitations of single-modal image-to-image image search in e-commerce by introducing MIEM, a multi-modal embedding model that fuses multiple product images with textual titles. MIEM employs a dual-tower architecture with a shared image encoder and a Merge Attention-based fusion module to produce robust item embeddings for recall, improving both recall and category accuracy while reducing storage compared to image-only approaches. The approach is trained on user click data with a three-stage pipeline and various losses (including AM-InfoNCE and modality Balance), and deployed with online/offline TensorRT-accelerated pipelines; online results show substantial gains in user engagement and conversion. Practically, MIEM enhances search relevance, reduces index size, and demonstrates real-world impact on Shopee’s image search, with future work exploring privacy-preserving enhancements.

Abstract

Over the past decade, significant advances have been made in the field of image search for e-commerce applications. Traditional image-to-image retrieval models, which focus solely on image details such as texture, tend to overlook useful semantic information contained within the images. As a result, the retrieved products might possess similar image details, but fail to fulfil the user's search goals. Moreover, the use of image-to-image retrieval models for products containing multiple images results in significant online product feature storage overhead and complex mapping implementations. In this paper, we report the design and deployment of the proposed Multi-modal Item Embedding Model (MIEM) to address these limitations. It is capable of utilizing both textual information and multiple images about a product to construct meaningful product features. By leveraging semantic information from images, MIEM effectively supplements the image search process, improving the overall accuracy of retrieval results. MIEM has become an integral part of the Shopee image search platform. Since its deployment in March 2023, it has achieved a remarkable 9.90% increase in terms of clicks per user and a 4.23% boost in terms of orders per user for the image search feature on the Shopee e-commerce platform.

Figures (6)

• Figure 1: Search results comparison using different models. The left column displays user queries as images, while the right column showcases the top 4 retrieved items. Each item image is accompanied by its respective product name. Notably, employing Image-to-image (I2I) solely emphasizes visual details, disregarding semantic information. However, by integrating MIEM with I2I, these limitations can be effectively addressed.
• Figure 2: An overview of the Shopee Image Search Engine.
• Figure 3: The integration of MIEM into the Shopee Image Search Engine. The arrows indicate the request triggering process.
• Figure 4: The system architecture of the AI Engine (MIEM). The yellow components are the trainable parts of the entire framework. The gray components are the training data.
• Figure 5: MIEM item embedding pre-calculation.