Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

From Visual to Multimodal: Systematic Ablation of Encoders and Fusion Strategies in Animal Identification

Vasiliy Kudryavtsev, Kirill Borodin, German Berezin, Kirill Bubenchikov, Grach Mkrtchian, Alexander Ryzhkov

TL;DR

A multimodal verification framework that enhances visual features with semantic identity priors derived from synthetic textual descriptions and demonstrates that integrating synthesized semantic descriptions significantly refines decision boundaries in large-scale pet re-identification.

Abstract

Automated animal identification is a practical task for reuniting lost pets with their owners, yet current systems often struggle due to limited dataset scale and reliance on unimodal visual cues. This study introduces a multimodal verification framework that enhances visual features with semantic identity priors derived from synthetic textual descriptions. We constructed a massive training corpus of 1.9 million photographs covering 695,091~unique animals to support this investigation. Through systematic ablation studies, we identified SigLIP2-Giant and E5-Small-v2 as the optimal vision and text backbones. We further evaluated fusion strategies ranging from simple concatenation to adaptive gating to determine the best method for integrating these modalities. Our proposed approach utilizes a gated fusion mechanism and achieved a Top-1 accuracy of 84.28\% and an Equal Error Rate of 0.0422 on a comprehensive test protocol. These results represent an 11\% improvement over leading unimodal baselines and demonstrate that integrating synthesized semantic descriptions significantly refines decision boundaries in large-scale pet re-identification.

From Visual to Multimodal: Systematic Ablation of Encoders and Fusion Strategies in Animal Identification

TL;DR

A multimodal verification framework that enhances visual features with semantic identity priors derived from synthetic textual descriptions and demonstrates that integrating synthesized semantic descriptions significantly refines decision boundaries in large-scale pet re-identification.

Abstract

Automated animal identification is a practical task for reuniting lost pets with their owners, yet current systems often struggle due to limited dataset scale and reliance on unimodal visual cues. This study introduces a multimodal verification framework that enhances visual features with semantic identity priors derived from synthetic textual descriptions. We constructed a massive training corpus of 1.9 million photographs covering 695,091~unique animals to support this investigation. Through systematic ablation studies, we identified SigLIP2-Giant and E5-Small-v2 as the optimal vision and text backbones. We further evaluated fusion strategies ranging from simple concatenation to adaptive gating to determine the best method for integrating these modalities. Our proposed approach utilizes a gated fusion mechanism and achieved a Top-1 accuracy of 84.28\% and an Equal Error Rate of 0.0422 on a comprehensive test protocol. These results represent an 11\% improvement over leading unimodal baselines and demonstrate that integrating synthesized semantic descriptions significantly refines decision boundaries in large-scale pet re-identification.
Paper Structure (31 sections, 9 equations, 10 figures, 26 tables)

This paper contains 31 sections, 9 equations, 10 figures, 26 tables.

Table of Contents

  1. Introduction
  2. Context and Relevance
  3. Research Problem and Associated Challenges
  4. Field Snapshot
  5. Gap and Rationale
  6. Contribution
  7. Materials and Methods
  8. Data
  9. Vision Encoder Experiments
  10. Text Generation
  11. Text Encoder Experiments
  12. Multimodal Experiments
  13. Comparison Methods
  14. Loss Function Design
  15. t-SNE Computation
  16. ...and 16 more sections

Figures (10)

  • Figure S1: Pet Identification Performance Across Training Data Configurations: ROC AUC, 1 - EER, and Top-k Metrics.
  • Figure S2: t-SNE visualization of training data embeddings under different data configurations.
  • Figure S3: Pet Identification Performance Across Vision Encoder Configurations: ROC AUC, 1 - EER, and Top-k Metrics.
  • Figure S4: t-SNE visualization of embeddings under different vision encoder configurations.
  • Figure S5: Pet Identification Performance Across Text Encoder Configurations: ROC AUC, 1 - EER, and Top-k Metrics.
  • ...and 5 more figures