When Reasoning Beats Scale: A 1.5B Reasoning Model Outranks 13B LLMs as Discriminator

TL;DR

The paper examines whether reasoning LLMs can serve as effective discriminators in a generator-discriminator planning framework for text-to-SQL, and introduces a method to derive fine-grained soft scores from chain-of-thought outputs. It systematically compares a distilled 1.5B reasoning model (DeepSeek-R1) against larger non-reasoning LLMs across intrinsic and end-to-end tasks and shows that the reasoning model can outperform larger non-reasoning models as a discriminator, while providing only modest gains from more context or compute. The study finds clear limits to the logical capabilities of reasoning models, with diminishing returns beyond moderate compute budgets and context, and reveals that generation remains harder than discrimination for these models. The results offer guidance for integrating reasoning models into planning infrastructures, suggesting they are most valuable as discriminators rather than generators in practical agentic systems.

Abstract

Large Language Models (LLM) with reasoning capabilities offer a promising path for improving candidate evaluation in planning frameworks, but their relative performance against traditional non-reasoning models remains largely underexplored. In this study, we benchmark a distilled 1.5B parameter reasoning model (DeepSeek-R1) against several state-of-the-art non-reasoning LLMs within a generator-discriminator LLM planning framework for the text-to-SQL task. For this, we introduce a novel method for extracting soft scores from the chain-of-thought (CoT) outputs from reasoning that enables fine-grained ranking of candidates. Our central hypothesis is that reasoning models are more effective discriminators than non-reasoning LLMs. Our results show that distilled DeepSeek-R1-1.5B achieves up to $87\%$ higher F1 and $3.7\%$ better discrimination accuracy than CodeLlama-7B, as well as $3.7\%$ higher execution accuracy than CodeLlama-13B, despite having significantly fewer parameters. Furthermore, we find that there is a limit to the logical capabilities of reasoning models, and only providing more context or allowing more compute budget for reasoning is not enough to improve their discrimination performance. Finally, we demonstrate that, unlike non-reasoning LLMs, reasoning models find generation more challenging than discrimination and may underperform as generators compared to smaller non-reasoning LLMs. Our work highlights the potential of reasoning models as discriminators in agentic frameworks, far outweighing their capabilities as generators, offering insights into their optimal role within LLM planning infrastructures.

Figures (5)

• Figure 1: Framework of LLM planning for Text-to-SQL task utilized for this study. For further information, see main.
• Figure 2: Flowchart illustrating our approach for obtaining soft evaluation scores from a reasoning model (Distill-R1) as a discriminator.
• Figure 3: Performance comparison of LLMs as discriminators in terms of execution accuracy during end-to-end evaluation (with Executability Check).
• Figure 4: Discrimination performance (top row) and reasoning quality (bottom row) of the reasoning model (Distill-R1) as a function of test-time compute budget (token limit).
• Figure 5: Performance comparison of LLMs as generators in terms of execution accuracy.