Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Heterogeneous Uncertainty-Guided Composed Image Retrieval with Fine-Grained Probabilistic Learning

Haomiao Tang, Jinpeng Wang, Minyi Zhao, Guanghao Meng, Ruisheng Luo, Long Chen, Shu-Tao Xia

TL;DR

This work tackles the heterogeneity and noise in Composed Image Retrieval by introducing Heterogeneous Uncertainty-Guided (HUG) learning. It represents each query and target with a sequence of Gaussian embeddings to capture fine-grained attributes and associated uncertainty, with separate estimators for target content quality and query multi-modal coordination. A provable dynamic weighting scheme fuses uncertainties across modalities, and the method optimizes both holistic and fine-grained uncertainty-guided contrasts to improve discriminative learning. Experiments on Fashion-IQ and CIRR demonstrate state-of-the-art performance and provide interpretable insights into how uncertainty components reflect attribute-level information and alignment quality, enabling robust search under noisy data conditions. The approach offers practical impact for user-centric visual search by effectively handling multi-modal ambiguity without requiring extra data or large language model prompts, and its theoretical guarantees reinforce the benefits of dynamic uncertainty fusion.

Abstract

Composed Image Retrieval (CIR) enables image search by combining a reference image with modification text. Intrinsic noise in CIR triplets incurs intrinsic uncertainty and threatens the model's robustness. Probabilistic learning approaches have shown promise in addressing such issues; however, they fall short for CIR due to their instance-level holistic modeling and homogeneous treatment of queries and targets. This paper introduces a Heterogeneous Uncertainty-Guided (HUG) paradigm to overcome these limitations. HUG utilizes a fine-grained probabilistic learning framework, where queries and targets are represented by Gaussian embeddings that capture detailed concepts and uncertainties. We customize heterogeneous uncertainty estimations for multi-modal queries and uni-modal targets. Given a query, we capture uncertainties not only regarding uni-modal content quality but also multi-modal coordination, followed by a provable dynamic weighting mechanism to derive comprehensive query uncertainty. We further design uncertainty-guided objectives, including query-target holistic contrast and fine-grained contrasts with comprehensive negative sampling strategies, which effectively enhance discriminative learning. Experiments on benchmarks demonstrate HUG's effectiveness beyond state-of-the-art baselines, with faithful analysis justifying the technical contributions.

Heterogeneous Uncertainty-Guided Composed Image Retrieval with Fine-Grained Probabilistic Learning

TL;DR

This work tackles the heterogeneity and noise in Composed Image Retrieval by introducing Heterogeneous Uncertainty-Guided (HUG) learning. It represents each query and target with a sequence of Gaussian embeddings to capture fine-grained attributes and associated uncertainty, with separate estimators for target content quality and query multi-modal coordination. A provable dynamic weighting scheme fuses uncertainties across modalities, and the method optimizes both holistic and fine-grained uncertainty-guided contrasts to improve discriminative learning. Experiments on Fashion-IQ and CIRR demonstrate state-of-the-art performance and provide interpretable insights into how uncertainty components reflect attribute-level information and alignment quality, enabling robust search under noisy data conditions. The approach offers practical impact for user-centric visual search by effectively handling multi-modal ambiguity without requiring extra data or large language model prompts, and its theoretical guarantees reinforce the benefits of dynamic uncertainty fusion.

Abstract

Composed Image Retrieval (CIR) enables image search by combining a reference image with modification text. Intrinsic noise in CIR triplets incurs intrinsic uncertainty and threatens the model's robustness. Probabilistic learning approaches have shown promise in addressing such issues; however, they fall short for CIR due to their instance-level holistic modeling and homogeneous treatment of queries and targets. This paper introduces a Heterogeneous Uncertainty-Guided (HUG) paradigm to overcome these limitations. HUG utilizes a fine-grained probabilistic learning framework, where queries and targets are represented by Gaussian embeddings that capture detailed concepts and uncertainties. We customize heterogeneous uncertainty estimations for multi-modal queries and uni-modal targets. Given a query, we capture uncertainties not only regarding uni-modal content quality but also multi-modal coordination, followed by a provable dynamic weighting mechanism to derive comprehensive query uncertainty. We further design uncertainty-guided objectives, including query-target holistic contrast and fine-grained contrasts with comprehensive negative sampling strategies, which effectively enhance discriminative learning. Experiments on benchmarks demonstrate HUG's effectiveness beyond state-of-the-art baselines, with faithful analysis justifying the technical contributions.
Paper Structure (31 sections, 2 theorems, 24 equations, 4 figures, 3 tables)

This paper contains 31 sections, 2 theorems, 24 equations, 4 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Composed Image Retrieval (CIR)
  4. Uncertainty Learning
  5. Our Solution
  6. Problem Formulation and Method Overview
  7. Heterogeneous Uncertainty Estimation
  8. Target Uncertainty regarding Visual Content Quality
  9. Query-Side Uncertainties regarding Uni-modal Quality and Multi-modal Coordination
  10. Summarized Query Uncertainty via Dynamic Weighting
  11. Uncertainty-Guided Learning
  12. Uncertainty-Guided Holistic Query-Target Contrast
  13. Uncertainty-Guided Fine-Grained Contrast
  14. Overall Learning Objectives
  15. Experiments
  16. ...and 16 more sections

Key Result

Proposition 1

Consider a loss function $\ell$ that is convex w.r.t.scalar variance values $\sigma_x^2,\,{x\!\in\!\{r,t,m\}}$. Given a training set $\mathcal{D}$ of size $N$, let $\hat{\mathbb{E}}[\ell(\sigma_x^2)] := \frac{1}{N} \sum_{n=1}^{N} \ell(\sigma_x^2(n))$ be the empirical estimate of the expected general where $\mathbb{E}(w_x)$ is the expectation of fusion weights, $\mathfrak{R}_x(\ell({\sigma_x^2}))$

Figures (4)

  • Figure 1: In Composed Image Retrieval, the uncertain multi-modal coordination between the reference image and modification text is also important in representation learning.
  • Figure 2: Heterogeneous Uncertainty-Guided (HUG) CIR. Modules with the same name share the same weights.
  • Figure 3: Model performance (average recall) on Fashion-IQ dataset under different settings of $\lambda_{\mathrm{Cord.}}$ and $\lambda_{\mathrm{FC}}$.
  • Figure 4: Qualitative analysis illustrating the meaning behind our learned uncertainty: (Left) Overall level of uncertainty reflects data quality. (Right) Different fine-grained uncertainty component corresponds to different sub-concepts.

Theorems & Definitions (6)

  • Proposition 1: Generalization Error Bounds
  • proof
  • Corollary 1
  • proof
  • proof
  • proof