Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Fine-Grained Classification: Connecting Metadata via Cross-Contrastive Pre-Training

Sumit Mamtani, Yash Thesia

TL;DR

This work tackles fine-grained visual classification by integrating appearance, descriptive class text, and spatio-temporal metadata into a unified $256$-D embedding through cross-contrastive pre-training. A six-term cross-modal objective enforces bidirectional alignment across image–text, image–metadata, and text–metadata, yielding transitive consistency that enhances discrimination among closely related species. After pre-training, a lightweight $512$-D image+metadata representation is fine-tuned with a two-layer classifier to predict $555$ NABirds classes, achieving $84.44\%$ top-1 on NABirds—$7.83\%$ higher than a vision-only baseline and surpassing GeoPrior. The approach demonstrates that geo-temporal context can resolve visually similar taxa and suggests broad applicability to domains where context is informative, though it requires reliable metadata and incurs additional pre-training cost.

Abstract

Fine-grained visual classification aims to recognize objects belonging to many subordinate categories of a supercategory, where appearance alone often fails to distinguish highly similar classes. We propose a unified framework that integrates image, text, and metadata via cross-contrastive pre-training. We first align the three modality encoders in a shared embedding space and then fine-tune the image and metadata encoders for classification. On NABirds, our approach improves over the baseline by 7.83% and achieves 84.44% top-1 accuracy, outperforming strong multimodal methods.

Fine-Grained Classification: Connecting Metadata via Cross-Contrastive Pre-Training

TL;DR

This work tackles fine-grained visual classification by integrating appearance, descriptive class text, and spatio-temporal metadata into a unified -D embedding through cross-contrastive pre-training. A six-term cross-modal objective enforces bidirectional alignment across image–text, image–metadata, and text–metadata, yielding transitive consistency that enhances discrimination among closely related species. After pre-training, a lightweight -D image+metadata representation is fine-tuned with a two-layer classifier to predict NABirds classes, achieving top-1 on NABirds— higher than a vision-only baseline and surpassing GeoPrior. The approach demonstrates that geo-temporal context can resolve visually similar taxa and suggests broad applicability to domains where context is informative, though it requires reliable metadata and incurs additional pre-training cost.

Abstract

Fine-grained visual classification aims to recognize objects belonging to many subordinate categories of a supercategory, where appearance alone often fails to distinguish highly similar classes. We propose a unified framework that integrates image, text, and metadata via cross-contrastive pre-training. We first align the three modality encoders in a shared embedding space and then fine-tune the image and metadata encoders for classification. On NABirds, our approach improves over the baseline by 7.83% and achieves 84.44% top-1 accuracy, outperforming strong multimodal methods.
Paper Structure (12 sections, 2 equations, 3 figures, 1 table)

This paper contains 12 sections, 2 equations, 3 figures, 1 table.

Table of Contents

  1. Introduction
  2. Related Work
  3. Proposed Model
  4. Cross-Contrastive Pre-Training
  5. Model Fine-Tuning
  6. Experiments
  7. Dataset
  8. Implementation Details
  9. Quantitative Evaluation
  10. Ablation of Cross-Contrastive Losses.
  11. Qualitative Evaluation
  12. Conclusion and Future Work

Figures (3)

  • Figure 1: Proposed architecture—pre-training of text, metadata, and image encoders using cross-contrastive loss.
  • Figure 2: Proposed architecture—fine-tuning and inference after pre-training using metadata and image encoders.
  • Figure 3: Qualitative Results : (a) Resulting embeddings for each input location from our model architecture trained on NABirds NAbird dataset. (b) Heat map of plausible locations for the European Starling given a fixed image and mid-year date.