Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Troika: Multi-Path Cross-Modal Traction for Compositional Zero-Shot Learning

Siteng Huang, Biao Gong, Yutong Feng, Min Zhang, Yiliang Lv, Donglin Wang

TL;DR

This work proposes a novel paradigm for CZSL models that establishes three identification branches (i.e., Multi-Path) to jointly model the state, object, and composition and presents Troika, an outstanding implementation that aligns the branch-specific prompt representations with decomposed visual features.

Abstract

Recent compositional zero-shot learning (CZSL) methods adapt pre-trained vision-language models (VLMs) by constructing trainable prompts only for composed state-object pairs. Relying on learning the joint representation of seen compositions, these methods ignore the explicit modeling of the state and object, thus limiting the exploitation of pre-trained knowledge and generalization to unseen compositions. With a particular focus on the universality of the solution, in this work, we propose a novel paradigm for CZSL models that establishes three identification branches (i.e., Multi-Path) to jointly model the state, object, and composition. The presented Troika is our implementation that aligns the branch-specific prompt representations with decomposed visual features. To calibrate the bias between semantically similar multi-modal representations, we further devise a Cross-Modal Traction module into Troika that shifts the prompt representation towards the current visual content. We conduct extensive experiments on three popular benchmarks, where our method significantly outperforms existing methods in both closed-world and open-world settings. The code will be available at https://github.com/bighuang624/Troika.

Troika: Multi-Path Cross-Modal Traction for Compositional Zero-Shot Learning

TL;DR

This work proposes a novel paradigm for CZSL models that establishes three identification branches (i.e., Multi-Path) to jointly model the state, object, and composition and presents Troika, an outstanding implementation that aligns the branch-specific prompt representations with decomposed visual features.

Abstract

Recent compositional zero-shot learning (CZSL) methods adapt pre-trained vision-language models (VLMs) by constructing trainable prompts only for composed state-object pairs. Relying on learning the joint representation of seen compositions, these methods ignore the explicit modeling of the state and object, thus limiting the exploitation of pre-trained knowledge and generalization to unseen compositions. With a particular focus on the universality of the solution, in this work, we propose a novel paradigm for CZSL models that establishes three identification branches (i.e., Multi-Path) to jointly model the state, object, and composition. The presented Troika is our implementation that aligns the branch-specific prompt representations with decomposed visual features. To calibrate the bias between semantically similar multi-modal representations, we further devise a Cross-Modal Traction module into Troika that shifts the prompt representation towards the current visual content. We conduct extensive experiments on three popular benchmarks, where our method significantly outperforms existing methods in both closed-world and open-world settings. The code will be available at https://github.com/bighuang624/Troika.
Paper Structure (47 sections, 18 equations, 10 figures, 16 tables)

This paper contains 47 sections, 18 equations, 10 figures, 16 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Compositional Zero-Shot Learning (CZSL).
  4. Parameter-Efficient Transfer Learning (PETL).
  5. Rethinking the Paradigm
  6. Preliminaries
  7. CZSL Task Formulation.
  8. Visual Feature Extraction.
  9. A Revisit of Existing Framework
  10. Generalizing in Multi-Path Paradigm
  11. Troika: An Efficient Implementation
  12. Instantiations
  13. Learning Prompt Representations.
  14. Learning Visual Representations.
  15. Training.
  16. ...and 32 more sections

Figures (10)

  • Figure 1: Graphical comparison of the existing paradigm and the proposed Multi-Path paradigm.
  • Figure 2: An example of the Cross-Modal Traction module. The commonly learned prompt of "red" may be further away (compared to "black") from individual images with the same concept, and the module reduces such mismatches by adaptively pulling the prompt representation towards the current visual content.
  • Figure 3: Graphical comparison of prompts constructed by prior methods and Troika.Red tokens are trainable.
  • Figure 4: Overview of the proposed Troika.
  • Figure 5: Visualization analysis of the Cross-Modal Traction module. We show the original image and the visualization result in pairs. The brighter the patch, the greater its role in the traction.
  • ...and 5 more figures