Learning Consistent Taxonomic Classification through Hierarchical Reasoning

TL;DR

This work tackles the problem of hierarchical inconsistency in taxonomic classification by Vision-Language Models. It introduces VL-Taxon, a two-stage hierarchical reasoning framework that first predicts the most specific leaf class via top-down reasoning and then refines all hierarchical levels conditioned on that leaf, aided by a hybrid SFT and GRPO training regimen. Experiments on iNaturalist-2021 demonstrate substantial gains in both leaf-level accuracy and hierarchical consistency, with the 7B backbone rivaling larger 72B models while using limited domain data. The approach also exhibits strong generalization to open-set and cross-domain taxa, highlighting its practical applicability for scalable, taxonomy-aware vision-language systems.

Abstract

While Vision-Language Models (VLMs) excel at visual understanding, they often fail to grasp hierarchical knowledge. This leads to common errors where VLMs misclassify coarser taxonomic levels even when correctly identifying the most specific level (leaf level). Existing approaches largely overlook this issue by failing to model hierarchical reasoning. To address this gap, we propose VL-Taxon, a two-stage, hierarchy-based reasoning framework designed to improve both leaf-level accuracy and hierarchical consistency in taxonomic classification. The first stage employs a top-down process to enhance leaf-level classification accuracy. The second stage then leverages this accurate leaf-level output to ensure consistency throughout the entire taxonomic hierarchy. Each stage is initially trained with supervised fine-tuning to instill taxonomy knowledge, followed by reinforcement learning to refine the model's reasoning and generalization capabilities. Extensive experiments reveal a remarkable result: our VL-Taxon framework, implemented on the Qwen2.5-VL-7B model, outperforms its original 72B counterpart by over 10% in both leaf-level and hierarchical consistency accuracy on average on the iNaturalist-2021 dataset. Notably, this significant gain was achieved by fine-tuning on just a small subset of data, without relying on any examples generated by other VLMs.

Figures (4)

• Figure 1: Left: Illustration of a typical plant taxonomic classification where VLMs are not able to follow the hierarchy even though their prediction at the most specific level (leaf level) is correct. Right: Examples of the predictions of the original Qwen2.5-VL-7B-Instruct and our extended VL-Taxon. $L$ denotes the level index in the test set. Correct/incorrect answers are colored green/red.
• Figure 2: Left: Example of inconsistent hierarchical listings across different levels for the same image, with correct/incorrect answers highlighted in green/red. Right: HCA across different levels' listing of the hierarchy. Blue lines indicate the HCA computed over all results at a given level, whereas orange lines represent the HCA computed only for cases where the leaf-level classification listed at a certain level is correct.
• Figure 3: Framework for the proposed VL-Taxon with two-stage hierarchical reasoning. Stage 1: Output the specific classification of the given image based on taxonomic hierarchical reasoning. Stage 2: Answer the specific question based on Stage 1's output to align the taxonomic hierarchy. The taxonomic hierarchical reasoning part in the thinking process is underlined in the example.
• Figure 4: Comparison of classification accuracy at each level on iNat21-Plant (L) and CUB-200 (R).