Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Risk Controlled Image Retrieval

Kaiwen Cai, Chris Xiaoxuan Lu, Xingyu Zhao, Xiaowei Huang

TL;DR

This work tackles the reliability gap in image retrieval by introducing Risk Controlled Image Retrieval (RCIR), which adds a probabilistic coverage guarantee to retrieved candidate sets. RCIR uses a Retrieval Set Size Adapter to scale the retrieval size based on query uncertainty and a Risk Controller to enforce a user-specified risk bound $\alpha$ with failure rate $\delta$ via a Hoeffding-based upper confidence bound. The approach yields provable guarantees regardless of data distribution or model choice, and experiments on CAR-196, CUB-200, Pittsburgh, and ChestX-Det demonstrate effective risk control with retrieval sets that adapt to query difficulty while remaining efficient. This framework enhances safety-critical retrieval applications by providing predictable coverage of true nearest neighbors and clarifies the role of uncertainty estimates in a probabilistic risk setting.

Abstract

Most image retrieval research prioritizes improving predictive performance, often overlooking situations where the reliability of predictions is equally important. The gap between model performance and reliability requirements highlights the need for a systematic approach to analyze and address the risks associated with image retrieval. Uncertainty quantification technique can be applied to mitigate this issue by assessing uncertainty for retrieval sets, but it provides only a heuristic estimate of uncertainty rather than a guarantee. To address these limitations, we present Risk Controlled Image Retrieval (RCIR), which generates retrieval sets with coverage guarantee, i.e., retrieval sets that are guaranteed to contain the true nearest neighbors with a predefined probability. RCIR can be easily integrated with existing uncertainty-aware image retrieval systems, agnostic to data distribution and model selection. To the best of our knowledge, this is the first work that provides coverage guarantees to image retrieval. The validity and efficiency of RCIR are demonstrated on four real-world datasets: CAR-196, CUB-200, Pittsburgh, and ChestX-Det.

Risk Controlled Image Retrieval

TL;DR

This work tackles the reliability gap in image retrieval by introducing Risk Controlled Image Retrieval (RCIR), which adds a probabilistic coverage guarantee to retrieved candidate sets. RCIR uses a Retrieval Set Size Adapter to scale the retrieval size based on query uncertainty and a Risk Controller to enforce a user-specified risk bound with failure rate via a Hoeffding-based upper confidence bound. The approach yields provable guarantees regardless of data distribution or model choice, and experiments on CAR-196, CUB-200, Pittsburgh, and ChestX-Det demonstrate effective risk control with retrieval sets that adapt to query difficulty while remaining efficient. This framework enhances safety-critical retrieval applications by providing predictable coverage of true nearest neighbors and clarifies the role of uncertainty estimates in a probabilistic risk setting.

Abstract

Most image retrieval research prioritizes improving predictive performance, often overlooking situations where the reliability of predictions is equally important. The gap between model performance and reliability requirements highlights the need for a systematic approach to analyze and address the risks associated with image retrieval. Uncertainty quantification technique can be applied to mitigate this issue by assessing uncertainty for retrieval sets, but it provides only a heuristic estimate of uncertainty rather than a guarantee. To address these limitations, we present Risk Controlled Image Retrieval (RCIR), which generates retrieval sets with coverage guarantee, i.e., retrieval sets that are guaranteed to contain the true nearest neighbors with a predefined probability. RCIR can be easily integrated with existing uncertainty-aware image retrieval systems, agnostic to data distribution and model selection. To the best of our knowledge, this is the first work that provides coverage guarantees to image retrieval. The validity and efficiency of RCIR are demonstrated on four real-world datasets: CAR-196, CUB-200, Pittsburgh, and ChestX-Det.
Paper Structure (24 sections, 2 theorems, 19 equations, 12 figures, 1 algorithm)

This paper contains 24 sections, 2 theorems, 19 equations, 12 figures, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Image Retrieval
  4. Uncertainty Estimation
  5. Conformal Prediction
  6. Method
  7. Image Retrieval
  8. Uncertainty-aware Image Retrieval
  9. Risk Controlled Image Retrieval
  10. Problem Setting
  11. Retrieval Set Size Adapter
  12. Risk Controller
  13. Experiments
  14. Datasets
  15. Implementations
  16. ...and 9 more sections

Key Result

Lemma 1

Given a dataset $\{\mathcal{Q}, \mathcal{D}\}$, with a fixed feature extractor $f_e$ and a fixed uncertainty estimator $f_u$, the risk function $\rho(\mathcal{R}_{[\kappa, f_{u}, f_{e}]})$ is a monotone nonincreasing function with respect to the scale $\kappa \in \mathds{R}^+$.

Figures (12)

  • Figure 1: Ilustration of our risk-controlled image retrieval RCIR and conventional uncertainty-aware image retrieval systems (e.g., warburg2021bayesian): An uncertainty-aware image retrieval provides a heuristic uncertainty, and has a fixed retrieval set size. In contrast, RCIR provides a retrieval set that is guaranteed to cover a true nearest neighbor with a user-specified risk requirement (probability of containing, and failure rate), and moreover the retrieval size adapts to the uncertainty of the query sample and the risk requirement.
  • Figure 2: Diagrams of a conventional image retrieval system and our RCIR: Blue blocks(i.e., Retrieval Set Size Adapter and Risk Controller) highlight our contributions. Please see the text below for a detailed explanation.
  • Figure 3: The $\rho-\kappa$ curve on the CAR-196 calibration set: the risk $\rho(\kappa)$ is monotone nonincreasing with $\kappa$.
  • Figure 4: The Recall@$1$ results of different methods on various test sets. The error bars represent the standard deviation of the results of $10$ trials.
  • Figure 5: The reliability diagrams on different test sets. The dashed line denotes the ideal calibration line. (The shadows represent the standard deviation of the results of $10$ trials, and the same goes for the other figures)
  • ...and 7 more figures

Theorems & Definitions (4)

  • Lemma 1
  • proof : Proof
  • Theorem 1
  • proof : Proof