Ada-R1: Hybrid-CoT via Bi-Level Adaptive Reasoning Optimization

TL;DR

The paper investigates the efficiency of long versus short chain-of-thought (CoT) reasoning in large language models and shows that the benefit of long, detailed reasoning is highly problem-dependent. It introduces Ada-R1, a two-stage adaptive framework that merges Long-CoT and Short-CoT capabilities into a single hybrid model and trains it with bi-level preferences to select a reasoning style at the group level and to compress reasoning within the chosen style at the instance level. Experiments on math benchmarks demonstrate substantial reductions in reasoning length (over 50% in several settings) with minimal or no loss in accuracy, outperforming baselines such as DPO and O1-Pruner. The results highlight the practical potential of adaptive reasoning to balance computational efficiency and performance in large-scale language models, particularly for complex problem solving.

Abstract

Recently, long-thought reasoning models achieve strong performance on complex reasoning tasks, but often incur substantial inference overhead, making efficiency a critical concern. Our empirical analysis reveals that the benefit of using Long-CoT varies across problems: while some problems require elaborate reasoning, others show no improvement, or even degraded accuracy. This motivates adaptive reasoning strategies that tailor reasoning depth to the input. However, prior work primarily reduces redundancy within long reasoning paths, limiting exploration of more efficient strategies beyond the Long-CoT paradigm. To address this, we propose a novel two-stage framework for adaptive and efficient reasoning. First, we construct a hybrid reasoning model by merging long and short CoT models to enable diverse reasoning styles. Second, we apply bi-level preference training to guide the model to select suitable reasoning styles (group-level), and prefer concise and correct reasoning within each style group (instance-level). Experiments demonstrate that our method (Ada-R1) significantly reduces inference costs compared to other baseline approaches, while maintaining performance. Notably, on five mathematical datasets, the average length of reasoning is reduced by more than 50%, highlighting the potential of adaptive strategies to optimize reasoning efficiency in large language models. Our code is coming soon at https://github.com/StarDewXXX/AdaR1

Figures (10)

• Figure 1: The proportion of gain in the data (left) and the relationship between CoT length and accuracy improvement (right), Long-CoT reasoning improves accuracy on difficult problems but has little effect or harms performance on easy ones.
• Figure 2: Pipeline of Ada-R1. At Stage I, we fused the models to obtain $\pi_{\theta_{H}}$. In Stage II, we sample from both long and short models and then elicit the group-level and instance-level preference. After this, we optimize $\pi_{\theta_{H}}$ at both group and instance level to obtain a hybrid adaptive reasoning model.
• Figure 3: The proportion and accuracy of thinking and non-thinking in different methods, Ada-R1 can achieve a good balance and accuracy between thinking and non-thinking.
• Figure 4: The ratio of thinking and non-thinking CoTs of Ada-R1-7B on different MATH levels (left) and the accuracy on different MATH levels of different models (right). As the difficulty increases, Ada-R1 is able to think more on harder problems and maintain higher accuracy.
• Figure 5: The ratio of accuracy at different MATH levels on different models. As the difficulty increases, Ada-R1 is able to maintain high accuracy.