Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Improving Visual Recommendation on E-commerce Platforms Using Vision-Language Models

Yuki Yada, Sho Akiyama, Ryo Watanabe, Yuta Ueno, Yusuke Shido, Andre Rusli

TL;DR

This work tackles the challenge of visual discovery on a large-scale C2C marketplace by fine-tuning a vision-language model (SigLIP) to generate image embeddings for visually similar product recommendations. The approach combines a ViT-B/16 image encoder with a multilingual text encoder, trained via sigmoid-based contrastive learning on $1$ million Mercari image-title pairs, and compressed to $128$-dimensional embeddings using PCA for efficient retrieval. Offline evaluation shows a $9.1\%$ gain in $nDCG@5$ and a $15.7\%$ gain in $Precision@1$ over a MobileNetV2 baseline, while online deployment yields a $50\%$ CTR uplift and a $14\%$ CVR uplift in live A/B tests. The production system features asynchronous embedding indexing, an ANN-based real-time search, and post-retrieval filtering and re-ranking, demonstrating that VLM-based encoders can meaningfully improve user engagement and conversion in e-commerce. The results motivate future work on integrating multimodal signals and personalized visual recommendations.

Abstract

On large-scale e-commerce platforms with tens of millions of active monthly users, recommending visually similar products is essential for enabling users to efficiently discover items that align with their preferences. This study presents the application of a vision-language model (VLM) -- which has demonstrated strong performance in image recognition and image-text retrieval tasks -- to product recommendations on Mercari, a major consumer-to-consumer marketplace used by more than 20 million monthly users in Japan. Specifically, we fine-tuned SigLIP, a VLM employing a sigmoid-based contrastive loss, using one million product image-title pairs from Mercari collected over a three-month period, and developed an image encoder for generating item embeddings used in the recommendation system. Our evaluation comprised an offline analysis of historical interaction logs and an online A/B test in a production environment. In offline analysis, the model achieved a 9.1% improvement in nDCG@5 compared with the baseline. In the online A/B test, the click-through rate improved by 50% whereas the conversion rate improved by 14% compared with the existing model. These results demonstrate the effectiveness of VLM-based encoders for e-commerce product recommendations and provide practical insights into the development of visual similarity-based recommendation systems.

Improving Visual Recommendation on E-commerce Platforms Using Vision-Language Models

TL;DR

This work tackles the challenge of visual discovery on a large-scale C2C marketplace by fine-tuning a vision-language model (SigLIP) to generate image embeddings for visually similar product recommendations. The approach combines a ViT-B/16 image encoder with a multilingual text encoder, trained via sigmoid-based contrastive learning on million Mercari image-title pairs, and compressed to -dimensional embeddings using PCA for efficient retrieval. Offline evaluation shows a gain in and a gain in over a MobileNetV2 baseline, while online deployment yields a CTR uplift and a CVR uplift in live A/B tests. The production system features asynchronous embedding indexing, an ANN-based real-time search, and post-retrieval filtering and re-ranking, demonstrating that VLM-based encoders can meaningfully improve user engagement and conversion in e-commerce. The results motivate future work on integrating multimodal signals and personalized visual recommendations.

Abstract

On large-scale e-commerce platforms with tens of millions of active monthly users, recommending visually similar products is essential for enabling users to efficiently discover items that align with their preferences. This study presents the application of a vision-language model (VLM) -- which has demonstrated strong performance in image recognition and image-text retrieval tasks -- to product recommendations on Mercari, a major consumer-to-consumer marketplace used by more than 20 million monthly users in Japan. Specifically, we fine-tuned SigLIP, a VLM employing a sigmoid-based contrastive loss, using one million product image-title pairs from Mercari collected over a three-month period, and developed an image encoder for generating item embeddings used in the recommendation system. Our evaluation comprised an offline analysis of historical interaction logs and an online A/B test in a production environment. In offline analysis, the model achieved a 9.1% improvement in nDCG@5 compared with the baseline. In the online A/B test, the click-through rate improved by 50% whereas the conversion rate improved by 14% compared with the existing model. These results demonstrate the effectiveness of VLM-based encoders for e-commerce product recommendations and provide practical insights into the development of visual similarity-based recommendation systems.
Paper Structure (13 sections, 3 figures, 1 table)

This paper contains 13 sections, 3 figures, 1 table.

Table of Contents

  1. Introduction
  2. Related Work
  3. Vision-Language Models
  4. Real-world Applications
  5. Visual Recommendation Using SigLIP
  6. Evaluation
  7. Model Training
  8. Offline Evaluation
  9. Model Deployment
  10. Online A/B Test
  11. Conclusion
  12. Authors Bio
  13. Acknowledgments

Figures (3)

  • Figure 1: Overview of a product recommendation system based on visual similarity.
  • Figure 2: Training pipeline. $I_1, \ldots, I_n$ represent the embedding vectors of product images 1 to $N$, and $T_1, \ldots, T_n$ represent the corresponding embedding vectors of product titles 1 to $N$.
  • Figure 3: System Architecture