Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

CREST: Cross-modal Resonance through Evidential Deep Learning for Enhanced Zero-Shot Learning

Haojian Huang, Xiaozhen Qiao, Zhuo Chen, Haodong Chen, Bingyu Li, Zhe Sun, Mulin Chen, Xuelong Li

TL;DR

CREST tackles the core challenge of zero-shot learning by aligning region-level visual features with semantic attributes under epistemic uncertainty. It introduces a bidirectional cross-modal framework (VGT/AGT) augmented with Evidential Deep Learning to quantify and fuse uncertainty across modalities, along with VICL and DIGS to stabilize region- and attribute-level grounding. The model incorporates ARISE to weave attribute semantics into the optimization objective and achieves strong CZSL/GZSL performance on CUB, SUN, and AWA2, with clear ablation and qualitative analyses demonstrating robustness and explainability. These contributions offer a scalable, interpretable approach to visual-semantic transfer in open-world settings, with potential for integration with large language models to further enhance semantic alignment.

Abstract

Zero-shot learning (ZSL) enables the recognition of novel classes by leveraging semantic knowledge transfer from known to unknown categories. This knowledge, typically encapsulated in attribute descriptions, aids in identifying class-specific visual features, thus facilitating visual-semantic alignment and improving ZSL performance. However, real-world challenges such as distribution imbalances and attribute co-occurrence among instances often hinder the discernment of local variances in images, a problem exacerbated by the scarcity of fine-grained, region-specific attribute annotations. Moreover, the variability in visual presentation within categories can also skew attribute-category associations. In response, we propose a bidirectional cross-modal ZSL approach CREST. It begins by extracting representations for attribute and visual localization and employs Evidential Deep Learning (EDL) to measure underlying epistemic uncertainty, thereby enhancing the model's resilience against hard negatives. CREST incorporates dual learning pathways, focusing on both visual-category and attribute-category alignments, to ensure robust correlation between latent and observable spaces. Moreover, we introduce an uncertainty-informed cross-modal fusion technique to refine visual-attribute inference. Extensive experiments demonstrate our model's effectiveness and unique explainability across multiple datasets. Our code and data are available at: https://github.com/JethroJames/CREST

CREST: Cross-modal Resonance through Evidential Deep Learning for Enhanced Zero-Shot Learning

TL;DR

CREST tackles the core challenge of zero-shot learning by aligning region-level visual features with semantic attributes under epistemic uncertainty. It introduces a bidirectional cross-modal framework (VGT/AGT) augmented with Evidential Deep Learning to quantify and fuse uncertainty across modalities, along with VICL and DIGS to stabilize region- and attribute-level grounding. The model incorporates ARISE to weave attribute semantics into the optimization objective and achieves strong CZSL/GZSL performance on CUB, SUN, and AWA2, with clear ablation and qualitative analyses demonstrating robustness and explainability. These contributions offer a scalable, interpretable approach to visual-semantic transfer in open-world settings, with potential for integration with large language models to further enhance semantic alignment.

Abstract

Zero-shot learning (ZSL) enables the recognition of novel classes by leveraging semantic knowledge transfer from known to unknown categories. This knowledge, typically encapsulated in attribute descriptions, aids in identifying class-specific visual features, thus facilitating visual-semantic alignment and improving ZSL performance. However, real-world challenges such as distribution imbalances and attribute co-occurrence among instances often hinder the discernment of local variances in images, a problem exacerbated by the scarcity of fine-grained, region-specific attribute annotations. Moreover, the variability in visual presentation within categories can also skew attribute-category associations. In response, we propose a bidirectional cross-modal ZSL approach CREST. It begins by extracting representations for attribute and visual localization and employs Evidential Deep Learning (EDL) to measure underlying epistemic uncertainty, thereby enhancing the model's resilience against hard negatives. CREST incorporates dual learning pathways, focusing on both visual-category and attribute-category alignments, to ensure robust correlation between latent and observable spaces. Moreover, we introduce an uncertainty-informed cross-modal fusion technique to refine visual-attribute inference. Extensive experiments demonstrate our model's effectiveness and unique explainability across multiple datasets. Our code and data are available at: https://github.com/JethroJames/CREST
Paper Structure (18 sections, 20 equations, 9 figures, 2 tables)

This paper contains 18 sections, 20 equations, 9 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Zero-shot learning
  4. Evidential Deep Learning for Classification
  5. Methodology
  6. Problem Definition
  7. Cross-modal Feature Extraction
  8. Visual Instance-level Contrastive Learning
  9. Decoupled Insight for Grounding Semantics
  10. Evidential deep learning
  11. Model training and optimization strategies
  12. Zero-Shot Inference
  13. Experiments
  14. Comparison with the State of the Art
  15. Ablation Studies
  16. ...and 3 more sections

Figures (9)

  • Figure 1: Challenges in instance-level recognition in the real world: (a) Attributes distribution imbalances—significant frequency differences among attributes; (b) Attributes co-occurrence—tendency of certain attributes to appear together, influencing model bias (further statistical details are available in the Supplementary Material).
  • Figure 2: The CREST model's architecture is depicted in Figure 2, initiating with modules (a) and (b) that perform bidirectional grounding to localize features within visuals and attributes. Following this, modules (c) and (d) engage in dual learning to align visual-category and attribute-category in the latent space. The process concludes with an uncertainty-driven fusion module (e), which integrates bidirectional evidence to enable robust visual-attribute inference.
  • Figure 3: The Birds of an identical category (i.e. black-footed albatross) captured in varying angles, backgrounds, distances, illumination, motions, etc. illustrating the dynamic nature of vision variability.
  • Figure 4: Illustration of attribute coupling across bird species, highlighting shared and divergent traits.
  • Figure 5: Visualization of Classification Confidence. In a three-category classification context, the correct outcome is presumed to be the first category. Ideally, a model with good calibration should yield Confident and Precise (CP) decisions (a) or Erroneous and Uncertain (EU) outcomes (d). On the other hand, instances of Confident but Unclear (CU) judgments (b) and Erroneous but Positive (EP) assertions (c) are indicative of areas where model certainty needs to be aligned more accurately with its precision.
  • ...and 4 more figures