Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Multi-Modality Transformer for E-Commerce: Inferring User Purchase Intention to Bridge the Query-Product Gap

Srivatsa Mallapragada, Ying Xie, Varsha Rani Chawan, Zeyad Hailat, Yuanbo Wang

TL;DR

The paper addresses the query–product gap in e-commerce search by modeling user purchase intention and leveraging granular product features from text and images. It introduces PINCER, a multi-modal transformer with a two-stage training pipeline that first learns purchase-intention vectors and relevance ranking, then generates pseudo-product embeddings via a transformer decoder that fuses purchase intention, product features, and query context. The method uses reward-based competitive learning, vector quantization, contrastive losses, and a preference modeling loss to align query, product, and intent spaces, achieving superior recall in real-world and synthetic experiments over strong baselines. The results demonstrate real-time retrieval capabilities and notable improvements in top-k recall, suggesting PINCER can effectively surface relevant products while capturing nuanced user intentions for scalable e-commerce search.

Abstract

E-commerce click-stream data and product catalogs offer critical user behavior insights and product knowledge. This paper propose a multi-modal transformer termed as PINCER, that leverages the above data sources to transform initial user queries into pseudo-product representations. By tapping into these external data sources, our model can infer users' potential purchase intent from their limited queries and capture query relevant product features. We demonstrate our model's superior performance over state-of-the-art alternatives on e-commerce online retrieval in both controlled and real-world experiments. Our ablation studies confirm that the proposed transformer architecture and integrated learning strategies enable the mining of key data sources to infer purchase intent, extract product features, and enhance the transformation pipeline from queries to more accurate pseudo-product representations.

Multi-Modality Transformer for E-Commerce: Inferring User Purchase Intention to Bridge the Query-Product Gap

TL;DR

The paper addresses the query–product gap in e-commerce search by modeling user purchase intention and leveraging granular product features from text and images. It introduces PINCER, a multi-modal transformer with a two-stage training pipeline that first learns purchase-intention vectors and relevance ranking, then generates pseudo-product embeddings via a transformer decoder that fuses purchase intention, product features, and query context. The method uses reward-based competitive learning, vector quantization, contrastive losses, and a preference modeling loss to align query, product, and intent spaces, achieving superior recall in real-world and synthetic experiments over strong baselines. The results demonstrate real-time retrieval capabilities and notable improvements in top-k recall, suggesting PINCER can effectively surface relevant products while capturing nuanced user intentions for scalable e-commerce search.

Abstract

E-commerce click-stream data and product catalogs offer critical user behavior insights and product knowledge. This paper propose a multi-modal transformer termed as PINCER, that leverages the above data sources to transform initial user queries into pseudo-product representations. By tapping into these external data sources, our model can infer users' potential purchase intent from their limited queries and capture query relevant product features. We demonstrate our model's superior performance over state-of-the-art alternatives on e-commerce online retrieval in both controlled and real-world experiments. Our ablation studies confirm that the proposed transformer architecture and integrated learning strategies enable the mining of key data sources to infer purchase intent, extract product features, and enhance the transformation pipeline from queries to more accurate pseudo-product representations.
Paper Structure (26 sections, 10 equations, 8 figures, 3 tables, 2 algorithms)

This paper contains 26 sections, 10 equations, 8 figures, 3 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Motivation 1: Purchase intention
  3. Motivation 2: Granular Product Features
  4. Proposed Multi-modal Transformer Framework
  5. Related Work
  6. Purchase intention estimation
  7. Granular product features in retrieval
  8. Multi-modal transformer for retrieval
  9. Methodology
  10. Model Architecture
  11. Stage 1: Purchase intention estimation and relevance ranking
  12. Stage 2: Pseudo product embedding generation
  13. Inference:
  14. Experiments
  15. Real-world experiments:
  16. ...and 11 more sections

Figures (8)

  • Figure 1: Examples to show the importance of purchase intention and product features in users' purchase choices from a ranked list of products
  • Figure 2: PINCER model architecture in various stages of training
  • Figure 3: Synthetic Add-To-Cart (ATC) data generation from FashionGen and Amazon datasets using a LLM
  • Figure 4: t-SNE distribution of FashionGen(mean gradient) randomly chosen query-product pairs
  • Figure 5: Retrieval comparison between RetroMAE, CLIP, and PINCER models for a query from Amazon (brightness) test dataset
  • ...and 3 more figures