Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Self-distilled Dynamic Fusion Network for Language-based Fashion Retrieval

Yiming Wu, Hangfei Li, Fangfang Wang, Yilong Zhang, Ronghua Liang

TL;DR

This paper tackles language-based fashion image retrieval by addressing the rigidity of static fusion with a Self-distilled Dynamic Fusion Network (SDFN). SDFN introduces Modality Specific Routers and a set of operation modules to dynamically fuse multi-granularity image and text features, reinforced by Self Path Distillation to stabilize routing decisions. The approach achieves state-of-the-art results on FashionIQ, Shoes, and Fashion200k benchmarks, highlighting strong gains from modality-aware routing, multi-module fusion, and consistency constraints. Overall, the method provides a flexible, robust framework for vision-language fashion retrieval with practical impact on cross-modal item search.

Abstract

In the domain of language-based fashion image retrieval, pinpointing the desired fashion item using both a reference image and its accompanying textual description is an intriguing challenge. Existing approaches lean heavily on static fusion techniques, intertwining image and text. Despite their commendable advancements, these approaches are still limited by a deficiency in flexibility. In response, we propose a Self-distilled Dynamic Fusion Network to compose the multi-granularity features dynamically by considering the consistency of routing path and modality-specific information simultaneously. Two new modules are included in our proposed method: (1) Dynamic Fusion Network with Modality Specific Routers. The dynamic network enables a flexible determination of the routing for each reference image and modification text, taking into account their distinct semantics and distributions. (2) Self Path Distillation Loss. A stable path decision for queries benefits the optimization of feature extraction as well as routing, and we approach this by progressively refine the path decision with previous path information. Extensive experiments demonstrate the effectiveness of our proposed model compared to existing methods.

Self-distilled Dynamic Fusion Network for Language-based Fashion Retrieval

TL;DR

This paper tackles language-based fashion image retrieval by addressing the rigidity of static fusion with a Self-distilled Dynamic Fusion Network (SDFN). SDFN introduces Modality Specific Routers and a set of operation modules to dynamically fuse multi-granularity image and text features, reinforced by Self Path Distillation to stabilize routing decisions. The approach achieves state-of-the-art results on FashionIQ, Shoes, and Fashion200k benchmarks, highlighting strong gains from modality-aware routing, multi-module fusion, and consistency constraints. Overall, the method provides a flexible, robust framework for vision-language fashion retrieval with practical impact on cross-modal item search.

Abstract

In the domain of language-based fashion image retrieval, pinpointing the desired fashion item using both a reference image and its accompanying textual description is an intriguing challenge. Existing approaches lean heavily on static fusion techniques, intertwining image and text. Despite their commendable advancements, these approaches are still limited by a deficiency in flexibility. In response, we propose a Self-distilled Dynamic Fusion Network to compose the multi-granularity features dynamically by considering the consistency of routing path and modality-specific information simultaneously. Two new modules are included in our proposed method: (1) Dynamic Fusion Network with Modality Specific Routers. The dynamic network enables a flexible determination of the routing for each reference image and modification text, taking into account their distinct semantics and distributions. (2) Self Path Distillation Loss. A stable path decision for queries benefits the optimization of feature extraction as well as routing, and we approach this by progressively refine the path decision with previous path information. Extensive experiments demonstrate the effectiveness of our proposed model compared to existing methods.
Paper Structure (11 sections, 7 equations, 2 figures, 3 tables)

This paper contains 11 sections, 7 equations, 2 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Methodology
  3. Image and Text Representation
  4. Operation Modules
  5. Modality Specific Routers
  6. Objective Function
  7. Experiments
  8. Experiment setting
  9. Main Results
  10. Ablation Studies
  11. Conclusion

Figures (2)

  • Figure 1: Difference between our proposed method and the existing composition methods.
  • Figure 2: Overall architecture of our proposed Self-distilled Dynamic Fusion Network. (a) Dynamic Fusion Network. (b) Modality Specific Routers. (c) Self-Path Distillation.