Bringing Multimodality to Amazon Visual Search System

TL;DR

The paper tackles street-to-shop visual search, where image-only matching often falters due to reliance on low-level features. It introduces Multimodal Image Matching (MIM) by incorporating vision-language alignment losses into deep metric learning, formulating 3-tower and 4-tower architectures that align query and catalog images with product text and, in the 4-tower case, short query text. Through extensive offline and online experiments, the authors demonstrate substantial improvements in recall and click-through rates, with the 4-tower setup offering additional gains over the 3-tower model. The work scales to Amazon-scale data (100M triples, 400M quadruples) and yields a practical multimodal search service that refines image-based results with textual reformulations, significantly enhancing search quality in production settings.

Abstract

Image to image matching has been well studied in the computer vision community. Previous studies mainly focus on training a deep metric learning model matching visual patterns between the query image and gallery images. In this study, we show that pure image-to-image matching suffers from false positives caused by matching to local visual patterns. To alleviate this issue, we propose to leverage recent advances in vision-language pretraining research. Specifically, we introduce additional image-text alignment losses into deep metric learning, which serve as constraints to the image-to-image matching loss. With additional alignments between the text (e.g., product title) and image pairs, the model can learn concepts from both modalities explicitly, which avoids matching low-level visual features. We progressively develop two variants, a 3-tower and a 4-tower model, where the latter takes one more short text query input. Through extensive experiments, we show that this change leads to a substantial improvement to the image to image matching problem. We further leveraged this model for multimodal search, which takes both image and reformulation text queries to improve search quality. Both offline and online experiments show strong improvements on the main metrics. Specifically, we see 4.95% relative improvement on image matching click through rate with the 3-tower model and 1.13% further improvement from the 4-tower model.

Figures (7)

• Figure 1: Street-to-shop problem: we use a lifestyle query image (left) to match product images (right) with simple or white background. Note the domain shift between the query and product images.
• Figure 2: Comparing image match results using traditional pure image-to-image loss versus the proposed method in this paper. Left: query images. Right: retrieved results. Row (a) shows the results from a traditional method and row (b) shows the results from a model trained with the proposed training paradigm. Red boxes mean wrong results. Blue boxes represent exact/similar matches to the query image. In the first query image, a customer tries to find a hair cut set. In the second query image, a customer tries to find an airpod case.
• Figure 3: Comparison between existing image match, proposed multimodal image match and multimodal search: (a) existing street-to-shop image matching is query image to product image matching; (b) we propose to use multimodal signals of the products for matching; (c) multimodal search where query image and reformulation text are used to perform the match.
• Figure 4: MIM diagram. We develop two variants (3-tower and 4-tower) of the MIM model. In the 3-tower architecture, we have two image encoders and one text encoder. The two image encoders are for query image and product image respectively, and the text encoder is for processing product text. The weights between two image encoders are shared. In the 4-tower architecture, we add one extra text encoder to process short text queries. The two text encoder weights are shared in the 4-tower architecture.
• Figure 5: Comparing street-to-shop retrieval from 4 representative models: (a) - (d) show results from Row 1, 2, 12, 13 respectively. Red boxes mean wrong results. Blue boxes represent exact/similar matches to the query image. The model trained with pure image-to-image match loss tends to match partial visual patterns while models trained with additional vision-text alignment loss can find the same or similar products from the same product category. The larger the model, the better the performance. We use our evaluation index to do the comparison, containing less than 1 percent of products compared to our online index, which makes exact matches hard.