Taxonomic Reasoning for Rare Arthropods: Combining Dense Image Captioning and RAG for Interpretable Classification

TL;DR

The paper tackles automated taxonomic classification of arthropods, especially rare species, by integrating dense image captioning with retrieval-augmented generation over external biodiversity texts. The two-module pipeline translates visual information into descriptive captions and then augments LLM reasoning with curated knowledge retrieves to produce taxonomic labels with calibrated confidence. Results show RAG-based reasoning improves classification for rare taxa and unknown species, while naive VLMs excel with familiar taxa; advanced retrieval offers limited gains, highlighting the need for richer knowledge bases. This approach promises scalable, interpretable biodiversity monitoring and conservation support, particularly when combined with citizen science data and ongoing knowledge base expansion.

Abstract

In the context of pressing climate change challenges and the significant biodiversity loss among arthropods, automated taxonomic classification from organismal images is a subject of intense research. However, traditional AI pipelines based on deep neural visual architectures such as CNNs or ViTs face limitations such as degraded performance on the long-tail of classes and the inability to reason about their predictions. We integrate image captioning and retrieval-augmented generation (RAG) with large language models (LLMs) to enhance biodiversity monitoring, showing particular promise for characterizing rare and unknown arthropod species. While a naive Vision-Language Model (VLM) excels in classifying images of common species, the RAG model enables classification of rarer taxa by matching explicit textual descriptions of taxonomic features to contextual biodiversity text data from external sources. The RAG model shows promise in reducing overconfidence and enhancing accuracy relative to naive LLMs, suggesting its viability in capturing the nuances of taxonomic hierarchy, particularly at the challenging family and genus levels. Our findings highlight the potential for modern vision-language AI pipelines to support biodiversity conservation initiatives, emphasizing the role of comprehensive data curation and collaboration with citizen science platforms to improve species identification, unknown species characterization and ultimately inform conservation strategies.

Figures (3)

• Figure 1: Example output from Module I (light blue) and Module II (light green) for an image of a yellow garden spider (Photo attribution: David Illig). In taxonomic classification: K=Kingdom, P=Phylum, C=Class, O=Order, F=Family, G=Genus. The RAG model generated accurate taxonomic labels to the genus level and refrained from providing a species level classification.
• Figure 2: Preprocessing of Wikipedia and Wikispecies text data includes chunking, filtering and contextualization before generating a vector database of embeddings. In-context images of Arthropoda are inputs to the RAG model and are initially fed through a VLM (Module I) to generate dense image biocaptions which are the subject of RAG queries to the vector database in Module II. The RAG model generates taxonomic classifications, accelerated biodiversity knowledge and commentary on LLM confidence for each image.
• Figure 3: Faithfulness and answer relevancy score distributions for accelerated biodiversity knowledge for Simple and Advanced RAG models. Medians (red lines), Inter quartile ranges (colour-filled boxes), maxima, minima as well as outliers more than 1.5 IQR outside the 1st and 3rd quartiles are shown. The Advanced RAG model uses MMR search criterion, multi-query and reranking while the Simple RAG model uses cosine similarity search criterion, no multiquery and no reranking.