Can Textual Reasoning Improve the Performance of MLLMs on Fine-grained Visual Classification?

TL;DR

This work shows that textual reasoning can degrade fine-grained visual perception in multimodal LLMs, a phenomenon termed the Cost of Thinking. It identifies that longer CoT sequences harm FGVC accuracy in zero-shot and that reinforcement fine-tuning drives a collapse toward concise reasoning, while still improving accuracy. To address this, the authors propose ReFine-RFT, a framework that combines ensemble rewards with Multi-Reward Normalization to constrain reasoning length while delivering dense accuracy feedback via $R_{cls}$, $R_{mllm}$, and $R_{emb}$. Across FGVC benchmarks, ReFine-RFT achieves state-of-the-art performance, especially when using parameter-efficient fine-tuning (LoRA), demonstrating that explicit reasoning length control and multi-objective reward shaping can enhance perception in MLLMs. The findings offer practical guidance for designing reward signals in multimodal reasoning systems and point to broader applications in perception-centric tasks.

Abstract

Multi-modal large language models (MLLMs) exhibit strong general-purpose capabilities, yet still struggle on Fine-Grained Visual Classification (FGVC), a core perception task that requires subtle visual discrimination and is crucial for many real-world applications. A widely adopted strategy for boosting performance on challenging tasks such as math and coding is Chain-of-Thought (CoT) reasoning. However, several prior works have reported that CoT can actually harm performance on visual perception tasks. These studies, though, examine the issue from relatively narrow angles and leave open why CoT degrades perception-heavy performance. We systematically re-examine the role of CoT in FGVC through the lenses of zero-shot evaluation and multiple training paradigms. Across these settings, we uncover a central paradox: the degradation induced by CoT is largely driven by the reasoning length, in which longer textual reasoning consistently lowers classification accuracy. We term this phenomenon the ``Cost of Thinking''. Building on this finding, we make two key contributions: (1) \alg, a simple and general plug-and-play normalization method for multi-reward optimization that balances heterogeneous reward signals, and (2) ReFine-RFT, a framework that combines ensemble rewards with \alg to constrain reasoning length while providing dense accuracy-oriented feedback. Extensive experiments demonstrate the effectiveness of our findings and the proposed ReFine-RFT, achieving state-of-the-art performance across FGVC benchmarks. Code and models are available at \href{https://github.com/jiezhu23/ReFine-RFT}{Project Link}.

Figures (8)

• Figure 1: Performance degradation with CoT and reasoning collapse in RFT. In zero-shot evaluation (top), MLLMs predict the correct label directly, but adding CoT reasoning leads to a wrong answer. During RFT (bottom), reasoning length steadily shrinks while accuracy improves, indicating a reasoning collapse.
• Figure 2: Dynamics of reasoning length during RFT across FGVC datasets. The dark green lines denote the running average of completion lengths throughout RFT FGVC tasks. Across all datasets, the reasoning content length rapidly decreases and stabilizes at a shorter range, suggesting that RFT discourages excessive reasoning generation and promotes concise, decision-focused responses. [Zero-shot: average content length of base model on the evaluation set; Step: the cumulative number of gradient update steps.]
• Figure 3: Impact of reasoning length on FGVC performance. We analyze the relationship between average reasoning (thinking) length and classification accuracy across FGVC datasets. As the average thinking length increases, performance consistently declines, indicating that excessive reasoning generation introduces noise or distracting the model from key discriminative visual cues.
• Figure 4: Overview of ReFine-RFT. Given a question, the model generates multiple candidate responses, each evaluated using an ensemble reward that combines rule-based rewards and model-based rewards like MLLM-based accuracy reward and embedding similarity reward. The proposed MRN then normalizes the rewards for each function to compute the final advantages used to update the MLLM.
• Figure 5: Differences among rewards during training. Each reward exhibits distinct convergence speed, value range, and saturation point, reflecting the heterogeneity of different rewards.