Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

UniHash: Unifying Pointwise and Pairwise Hashing Paradigms for Seen and Unseen Category Retrieval

Xiaoxu Ma, Runhao Li, Hanwen Liu, Xiangbo Zhang, Zhenyu Weng

TL;DR

UniHash tackles the challenge of retrieving images across seen and unseen categories by unifying center-based pointwise and pairwise hashing into a dual-branch framework. It introduces a mutual learning loss and a Split-Merge Mixture of Hash Experts (SM-MoH) to enable deep cross-branch knowledge exchange, producing discriminative and generalizable $q$-bit hash codes. Theoretical analysis shows that UniHash converts paradigm-specific error floors into a vanishing consistency term, improving seen/unseen generalization. Empirically, UniHash achieves state-of-the-art performance on CIFAR-10, ImageNet, and MSCOCO under both closed-set and seen/unseen protocols, demonstrating strong robustness and practical impact for large-scale and open-set image retrieval.

Abstract

Effective retrieval across both seen and unseen categories is crucial for modern image retrieval systems. Retrieval on seen categories ensures precise recognition of known classes, while retrieval on unseen categories promotes generalization to novel classes with limited supervision. However, most existing deep hashing methods are confined to a single training paradigm, either pointwise or pairwise, where the former excels on seen categories and the latter generalizes better to unseen ones. To overcome this limitation, we propose Unified Hashing (UniHash), a dual-branch framework that unifies the strengths of both paradigms to achieve balanced retrieval performance across seen and unseen categories. UniHash consists of two complementary branches: a center-based branch following the pointwise paradigm and a pairwise branch following the pairwise paradigm. A novel hash code learning method is introduced to enable bidirectional knowledge transfer between branches, improving hash code discriminability and generalization. It employs a mutual learning loss to align hash representations and introduces a Split-Merge Mixture of Hash Experts (SM-MoH) module to enhance cross-branch exchange of hash representations. Theoretical analysis substantiates the effectiveness of UniHash, and extensive experiments on CIFAR-10, MSCOCO, and ImageNet demonstrate that UniHash consistently achieves state-of-the-art performance in both seen and unseen image retrieval scenarios.

UniHash: Unifying Pointwise and Pairwise Hashing Paradigms for Seen and Unseen Category Retrieval

TL;DR

UniHash tackles the challenge of retrieving images across seen and unseen categories by unifying center-based pointwise and pairwise hashing into a dual-branch framework. It introduces a mutual learning loss and a Split-Merge Mixture of Hash Experts (SM-MoH) to enable deep cross-branch knowledge exchange, producing discriminative and generalizable -bit hash codes. Theoretical analysis shows that UniHash converts paradigm-specific error floors into a vanishing consistency term, improving seen/unseen generalization. Empirically, UniHash achieves state-of-the-art performance on CIFAR-10, ImageNet, and MSCOCO under both closed-set and seen/unseen protocols, demonstrating strong robustness and practical impact for large-scale and open-set image retrieval.

Abstract

Effective retrieval across both seen and unseen categories is crucial for modern image retrieval systems. Retrieval on seen categories ensures precise recognition of known classes, while retrieval on unseen categories promotes generalization to novel classes with limited supervision. However, most existing deep hashing methods are confined to a single training paradigm, either pointwise or pairwise, where the former excels on seen categories and the latter generalizes better to unseen ones. To overcome this limitation, we propose Unified Hashing (UniHash), a dual-branch framework that unifies the strengths of both paradigms to achieve balanced retrieval performance across seen and unseen categories. UniHash consists of two complementary branches: a center-based branch following the pointwise paradigm and a pairwise branch following the pairwise paradigm. A novel hash code learning method is introduced to enable bidirectional knowledge transfer between branches, improving hash code discriminability and generalization. It employs a mutual learning loss to align hash representations and introduces a Split-Merge Mixture of Hash Experts (SM-MoH) module to enhance cross-branch exchange of hash representations. Theoretical analysis substantiates the effectiveness of UniHash, and extensive experiments on CIFAR-10, MSCOCO, and ImageNet demonstrate that UniHash consistently achieves state-of-the-art performance in both seen and unseen image retrieval scenarios.
Paper Structure (34 sections, 2 theorems, 29 equations, 7 figures, 6 tables, 1 algorithm)

This paper contains 34 sections, 2 theorems, 29 equations, 7 figures, 6 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Works
  3. Methodology
  4. UniHash Framework
  5. Mutual Learning Loss
  6. SM-MoH Module
  7. Discrepancy Elimination Theory
  8. Preliminaries: Assumptions and Capacity Terms
  9. UniHash Superiority Analysis
  10. Experiments
  11. Closed-Set Retrieval Performance
  12. Generalization to Unseen Categories
  13. Ablation Study
  14. Hash Codes Visualization
  15. Conclusions
  16. ...and 19 more sections

Key Result

Proposition 1

For a single-branch hashing method, the population risk obeys where $E_{struct}>0$ denotes a paradigm-specific, irreducible discrepancy independent of both $n$ and $q$. Specifically, a center-based method, which lacks relational constraints, incurs a relative discrepancy $E_{\mathrm{rel}}>0$, whereas a pairwise method, lacking global semantic anchors, suffers

Figures (7)

  • Figure 1: Comparison of existing deep hashing paradigms with the proposed Unified Hashing (UniHash). The figure depicts the evolution from traditional center-based and pairwise hashing to the proposed unified hashing paradigm, which fosters complementary supervision between branches and enhances retrieval performance on both seen and unseen image categories.
  • Figure 2: Overview of the proposed Unified Hashing (UniHash) framework. A deep neural network extracts image features, which are fed into two parallel branches producing center-based codes $u_c$ and pairwise codes $u_p$. During training, the center-based branch learns relational cues from the pairwise branch, while the pairwise branch benefits from the semantic compactness of the center-based branch, leading to more discriminative and generalizable hash codes.
  • Figure 3: Comparison of expert-based hashing architectures. (a) Conventional MoE with separate experts per branch. (b) Only-Split MoH with independent gates and experts generating continuous hash codes. (c) Our proposed MoH with independent gates and shared experts.
  • Figure 4: Precision-Recall curves on ImageNet across different bit configurations.
  • Figure 5: Impact of Unified Hashing (UniHash) on hash code distribution on ImageNet with 16-bit configuration.
  • ...and 2 more figures

Theorems & Definitions (3)

  • Proposition 1: Irreducible Structural Discrepancy
  • Theorem 1: Discrepancy Elimination via UniHash
  • proof