Fine-R1: Make Multi-modal LLMs Excel in Fine-Grained Visual Recognition by Chain-of-Thought Reasoning

TL;DR

Fine-R1 tackles the data inefficiency and base-to-new generalization gaps in fine-grained visual recognition by coupling Chain-of-Thought supervised fine-tuning with Triplet Augmented Policy Optimization. The two-stage approach first imbues MLLMs with structured FGVR reasoning and then optimizes discriminative deployment under intra-class and inter-class variation using few-shot data. Across six FGVR datasets in a 4-shot base-to-new setup, Fine-R1 achieves state-of-the-art results in both closed-world and open-world settings, outperforming general MLLMs, reasoning-focused MLLMs, and even CLIP models. Analyses indicate improvements arise from better deployment of existing fine-grained knowledge rather than wholesale changes to visual features, signaling strong potential for knowledge-intensive FGVR domains.

Abstract

Any entity in the visual world can be hierarchically grouped based on shared characteristics and mapped to fine-grained sub-categories. While Multi-modal Large Language Models (MLLMs) achieve strong performance on coarse-grained visual tasks, they often struggle with Fine-Grained Visual Recognition (FGVR). Adapting general-purpose MLLMs to FGVR typically requires large amounts of annotated data, which is costly to obtain, leaving a substantial performance gap compared to contrastive CLIP models dedicated for discriminative tasks. Moreover, MLLMs tend to overfit to seen sub-categories and generalize poorly to unseen ones. To address these challenges, we propose Fine-R1, an MLLM tailored for FGVR through an R1-style training framework: (1) Chain-of-Thought Supervised Fine-tuning, where we construct a high-quality FGVR CoT dataset with rationales of "visual analysis, candidate sub-categories, comparison, and prediction", transition the model into a strong open-world classifier; and (2) Triplet Augmented Policy Optimization, where Intra-class Augmentation mixes trajectories from anchor and positive images within the same category to improve robustness to intra-class variance, while Inter-class Augmentation maximizes the response distinction conditioned on images across sub-categories to enhance discriminative ability. With only 4-shot training, Fine-R1 outperforms existing general MLLMs, reasoning MLLMs, and even contrastive CLIP models in identifying both seen and unseen sub-categories, showing promise in working in knowledge-intensive domains where gathering expert annotations for all sub-categories is arduous. Code is available at https://github.com/PKU-ICST-MIPL/FineR1_ICLR2026.

Figures (7)

• Figure 1: Fine-R1 generates Chain-of-Thought (CoT) before producing the final fine-grained visual recognition (FGVR) answer. It utilizes CoT supervised fine-tuning (SFT) and Triplet Augmented Policy Optimization (TAPO), learning the reasoning process with only few-shot samples per category. In comparison to general and reasoning MLLMs, and contrastive CLIP models, Fine-R1 excels in identifying both seen and unseen categories.
• Figure 2: Overview of the proposed two-stage training framework integrating CoT SFT and TAPO.
• Figure 3: Ablation study on training methods, inference strategies, and key components of Fine-R1.
• Figure 3: Ablation study on $n_1$:$n_2$, #CoTs in stage 1, and cross-model evaluation.
• Figure 4: PCA projections of the last hidden state representations of inputs containing positive and negative image-category pairs, extracted from Qwen2.5-VL-3B and Fine-R1.