Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Beyond Text: Aligning Vision and Language for Multimodal E-Commerce Retrieval

Qujiaheng Zhang, Guagnyue Xu, Fengjie Li

TL;DR

This work proposes a noval modality fusion network to fuse image and text information and capture cross-modal complementary information in the e-commerce domain and demonstrates that domain-specific fine-tuning and two stage alignment between query with product text and image modalities are both crucial for effective multimodal retrieval.

Abstract

Modern e-commerce search is inherently multimodal: customers make purchase decisions by jointly considering product text and visual informations. However, most industrial retrieval and ranking systems primarily rely on textual information, underutilizing the rich visual signals available in product images. In this work, we study unified text-image fusion for two-tower retrieval models in the e-commerce domain. We demonstrate that domain-specific fine-tuning and two stage alignment between query with product text and image modalities are both crucial for effective multimodal retrieval. Building on these insights, we propose a noval modality fusion network to fuse image and text information and capture cross-modal complementary information. Experiments on large-scale e-commerce datasets validate the effectiveness of the proposed approach.

Beyond Text: Aligning Vision and Language for Multimodal E-Commerce Retrieval

TL;DR

This work proposes a noval modality fusion network to fuse image and text information and capture cross-modal complementary information in the e-commerce domain and demonstrates that domain-specific fine-tuning and two stage alignment between query with product text and image modalities are both crucial for effective multimodal retrieval.

Abstract

Modern e-commerce search is inherently multimodal: customers make purchase decisions by jointly considering product text and visual informations. However, most industrial retrieval and ranking systems primarily rely on textual information, underutilizing the rich visual signals available in product images. In this work, we study unified text-image fusion for two-tower retrieval models in the e-commerce domain. We demonstrate that domain-specific fine-tuning and two stage alignment between query with product text and image modalities are both crucial for effective multimodal retrieval. Building on these insights, we propose a noval modality fusion network to fuse image and text information and capture cross-modal complementary information. Experiments on large-scale e-commerce datasets validate the effectiveness of the proposed approach.
Paper Structure (21 sections, 8 equations, 2 figures, 3 tables)

This paper contains 21 sections, 8 equations, 2 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Proposed Method
  3. Task Formulation
  4. Model Architecture
  5. Query and Item Encoding.
  6. Mixture-of-Modality Fusion.
  7. Cross-modal Interaction.
  8. Self-Adversarial Negative Sampling
  9. Training Objective
  10. Curriculum Training Strategy
  11. Experiments
  12. Training Dataset
  13. Evaluation Dataset
  14. Experimental Setup
  15. Metrics
  16. ...and 6 more sections

Figures (2)

  • Figure 1: Home decor search is highly visual: product images provide critical relevance signals beyond text, motivating multimodal retrieval.
  • Figure 2: Proposed multimodal retrieval framework. Left: curriculum training procedure for domain adaptation and query alignment; Middle: multimodal two-tower retrieval architecture; Right: bilinear block architecture.