Adaptive-Solver Framework for Dynamic Strategy Selection in Large Language Model Reasoning

TL;DR

The paper presents Adaptive-Solver (AS), a dynamic framework that selects solving strategies for LLM reasoning to balance accuracy and test-time cost. It combines an evaluation module (consistency-based quality check) with an adaptation module (four strategies: model, sample size, prompting method, and decomposition granularity) and a pipeline configuration algorithm to assemble effective solver sequences. Empirical results across eight reasoning datasets show substantial API-cost reductions (up to 85%) while maintaining or surpassing baselines at equivalent budgets, with insights into how each adaptation strategy contributes to performance and cost. The approach demonstrates that multi-dimensional solver adaptation and automated pipeline design can substantially improve the efficiency and robustness of LLM reasoning in practice.

Abstract

Large Language Models (LLMs) demonstrate impressive ability in handling reasoning tasks. However, unlike humans who can instinctively adapt their problem-solving strategies to the complexity of task, most LLM-based methods adopt a one-size-fits-all approach. These methods employ consistent models, sample sizes, prompting methods and levels of problem decomposition, regardless of the problem complexity. The inflexibility of these methods can bring unnecessary computational overhead or sub-optimal performance. To address this limitation, we introduce an Adaptive-Solver (AS) framework tha dynamically adapts solving strategies to suit various problems, enabling the flexible allocation of test-time computational resources. The framework functions with two primary modules. The initial evaluation module assesses the reliability of the current solution using answer consistency. If the solution is deemed unreliable, the subsequent adaptation module comes into play. Within this module, various types of adaptation strategies are employed collaboratively. Through such dynamic and multi-faceted adaptations, our framework can help reduce computational consumption and improve performance. Experimental results from complex reasoning benchmarks reveal that our method can significantly reduce API costs (up to 85%) while maintaining original performance. Alternatively, it achieves up to 4.5% higher accuracy compared to the baselines at the same cost. The code and dataset are available at https://github.com/john1226966735/Adaptive-Solver.

Figures (23)

• Figure 1: A motivation illustration. Difficulty is measured by the number of steps in the ground-truth solution. (a) The performance advantage of a larger, more expensive model over a smaller, cheaper model varies across datasets; for simpler tasks, smaller models can perform comparably to larger ones. (b) Different prompting methods have unique strengths, so the optimal prompting approach depends on the characteristics of each dataset. (c) For tasks of varying difficulty, particularly simpler ones, using a smaller sample size can achieve similar accuracy as a larger sample size while reducing costs. (d) For tasks with different difficulty levels, the ideal decomposition granularity varies.
• Figure 2: Comparison of the frameworks of our method and baselines. (a) Existing methods utilize static solvers. (b) Our framework selects a suitable solver from candidate solvers for each different problem. The red section highlights the differences between our method and the baselines.
• Figure 3: Overview of the Adaptive-Solver framework. It consists of two main modules: the evaluation module assesses if the current solution meets the required criteria; if not, the adaptation module adjusts the current solver by selecting the next solver from a predetermined pipeline. For simplicity, we illustrate a scenario with two solving rounds.
• Figure 4: Illustration of the four adaptation strategies. These strategies respectively consider the perspectives of the LLM model, sample size, prompt, and decomposition granularity.
• Figure 5: Illustration of the process of pipeline configuration.