Evaluating Large Language Models for IUCN Red List Species Information

TL;DR

This study benchmarks five state-of-the-art LLMs across 21,955 species to assess four IUCN Red List components: taxonomy, conservation status, distribution, and threats. It reveals a pronounced dichotomy: LLMs achieve near-perfect taxonomy classification (~95%) yet struggle with ecological judgment tasks (≈27% for Red List status), with systematic biases favoring vertebrates. The results support a hybrid AI-human workflow where LLMs expedite information retrieval and evidence extraction while humans make threshold-based and causal judgments, aided by assessor-facing tools and standardized evaluation frameworks. The work also highlights biases in biodiversity knowledge bases and calls for taxonomically stratified deployment and multilingual data integration to ensure equitable conservation outcomes.

Abstract

Large Language Models (LLMs) are rapidly being adopted in conservation to address the biodiversity crisis, yet their reliability for species evaluation is uncertain. This study systematically validates five leading models on 21,955 species across four core IUCN Red List assessment components: taxonomy, conservation status, distribution, and threats. A critical paradox was revealed: models excelled at taxonomic classification (94.9%) but consistently failed at conservation reasoning (27.2% for status assessment). This knowledge-reasoning gap, evident across all models, suggests inherent architectural constraints, not just data limitations. Furthermore, models exhibited systematic biases favoring charismatic vertebrates, potentially amplifying existing conservation inequities. These findings delineate clear boundaries for responsible LLM deployment: they are powerful tools for information retrieval but require human oversight for judgment-based decisions. A hybrid approach is recommended, where LLMs augment expert capacity while human experts retain sole authority over risk assessment and policy.

Figures (7)

• Figure 1: Alluvial diagram of IUCN Red List category predictions aggregated across all evaluated LLMs (n = 109,755 predictions from 21,951 species with valid Red List status). The diagram links target categories (left) with predicted categories (right), with flow thickness proportional to the number of predictions. Vertical flows represent correct classifications, while diagonal flows indicate misclassifications. Most errors occur between adjacent categories (e.g., EN and VU: 2,862 cases; NT and LC: 3,866 cases), indicating that LLMs tend to confuse neighboring threat levels rather than making random errors. This suggests that the models capture the ordinal structure of Red List categories but struggle with applying precise thresholds.
• Figure 2: Heatmap of task-specific prediction accuracy by taxonomic group across five LLMs. Cell values indicate mean accuracy (%) with color intensity proportional to accuracy. Vertebrates consistently achieve the highest accuracy across all tasks (mean: 62.3%), while fungi show the lowest (mean: 45.8%). The variation across groups is minimal for taxonomic classification (range: 87.5--95.6%) but substantial for conservation-specific tasks. For instance, in Red List category assessment, mammals (50.8%) outperform amphibians (33.5%) by 17.3 percentage points.
• Figure 3: System message template for Task 1 (Taxonomic Classification). This prompt instructs the model to act as a biological taxonomy expert and specifies the required output format combining JSON classification with multiple-choice answer selection.
• Figure 4: Question template for Task 1 showing the minimal format used to reduce token consumption while maintaining task clarity. Variables question and choices are dynamically populated for each species.
• Figure 5: System message template for Task 2 (Red List Category Assessment) defining all nine IUCN categories with their codes and providing examples of expected input-output format.