Neural Catalog: Scaling Species Recognition with Catalog of Life-Augmented Generation

TL;DR

This work tackles open-vocabulary species recognition by introducing VR-RAG, a Retrieval-augmented Generation framework with a visual re-ranking step that fuses structured encyclopedic knowledge with vision-language reasoning. It builds a large, open-world knowledge base by distilling Wikipedia descriptions for $11{,}202$ bird species and curating a Pokémon dataset, then retrieves and refines candidate species using a multimodal, re-ranked pipeline. The approach yields substantial gains in retrieval (mRR) and open-vocabulary recognition accuracy across five bird benchmarks, FishNet, and Pokémon, achieving an average improvement of $18.0\%$ over the strongest MLLMs and notable improvements over baselines like CLIP/OpenCLIP/SigLIP. By enabling high-quality reasoning with concise, discriminative summaries and avoiding extensive retraining, VR-RAG demonstrates strong cross-domain applicability for fine-grained recognition in open-world settings.

Abstract

Open-vocabulary species recognition is a major challenge in computer vision, particularly in ornithology, where new taxa are continually discovered. While benchmarks like CUB-200-2011 and Birdsnap have advanced fine-grained recognition under closed vocabularies, they fall short of real-world conditions. We show that current systems suffer a performance drop of over 30\% in realistic open-vocabulary settings with thousands of candidate species, largely due to an increased number of visually similar and semantically ambiguous distractors. To address this, we propose Visual Re-ranking Retrieval-Augmented Generation (VR-RAG), a novel framework that links structured encyclopedic knowledge with recognition. We distill Wikipedia articles for 11,202 bird species into concise, discriminative summaries and retrieve candidates from these summaries. Unlike prior text-only approaches, VR-RAG incorporates visual information during retrieval, ensuring final predictions are both textually relevant and visually consistent with the query image. Extensive experiments across five bird classification benchmarks and two additional domains show that VR-RAG improves the average performance of the state-of-the-art Qwen2.5-VL model by 18.0%.

Figures (16)

• Figure 1: The VR-RAG pipeline. Left: Data extraction, both textual and visual, from Wikipedia articles. Middle: Similarity calculation with the query image using an ensemble of multi-modal encoders. Right: Re-ranker module using anchor images for final similarity score calculation.
• Figure 2: Impact of varying top-k candidates($5$ to $15$) fed to Qwen2.5-VL.
• Figure 3: Impact of varying $\lambda$.
• Figure 4: Effect of re-ranking on retrieval performance. Average mRR at ranks 1, 5, and 10 is shown as a function of the number of candidates re-ranked. Performance improves substantially when re-ranking the first few dozen candidates, but the gains plateau quickly. Beyond 100 candidates, the improvement is marginal, motivating the choice of 100 as the default setting in the main experiments.
• Figure 5: Top-k species selection ablation. We report the results for Gemma-3n on all five benchmarks by varying the value of k from 5 to 15.