A Comprehensive Evaluation of LLM Reasoning: From Single-Model to Multi-Agent Paradigms

TL;DR

We address how different LLM reasoning paradigms—direct single-model generation, CoT-augmented reasoning, and MAS workflows—perform under practical cost and accuracy constraints. The authors introduce a unified evaluation framework and the MIMeBench benchmark to assess both final-answer accuracy and foundational semantic skills like abstraction and contrastive discrimination. Across closed-form benchmarks and open-ended evaluation, results show that increasing structural complexity does not guarantee improvements; gains are highly task-dependent and influenced by specific paradigm dynamics such as reflection and internal/external deliberation. The work provides practical guidance for deploying LLM-based reasoning systems and supplies open benchmarks and code to foster further research.

Abstract

Large Language Models (LLMs) are increasingly deployed as reasoning systems, where reasoning paradigms - such as Chain-of-Thought (CoT) and multi-agent systems (MAS) - play a critical role, yet their relative effectiveness and cost-accuracy trade-offs remain poorly understood. In this work, we conduct a comprehensive and unified evaluation of reasoning paradigms, spanning direct single-model generation, CoT-augmented single-model reasoning, and representative MAS workflows, characterizing their reasoning performance across a diverse suite of closed-form benchmarks. Beyond overall performance, we probe role-specific capability demands in MAS using targeted role isolation analyses, and analyze cost-accuracy trade-offs to identify which MAS workflows offer a favorable balance between cost and accuracy, and which incur prohibitive overhead for marginal gains. We further introduce MIMeBench, a new open-ended benchmark that targets two foundational yet underexplored semantic capabilities - semantic abstraction and contrastive discrimination - thereby providing an alternative evaluation axis beyond closed-form accuracy and enabling fine-grained assessment of semantic competence that is difficult to capture with existing benchmarks. Our results show that increased structural complexity does not consistently lead to improved reasoning performance, with its benefits being highly dependent on the properties and suitability of the reasoning paradigm itself. The codes are released at https://gitcode.com/HIT1920/OpenLLMBench.

Figures (21)

• Figure 1: Overview of our study. We evaluate multiple reasoning paradigms under a unified protocol using closed-form benchmarks (left), and complement them with an open-ended benchmark, MIMeBench (right).
• Figure 2: Overview of MAS Workflows.
• Figure 3: Mean token cost and success rate across MAS workflows on ARC-Challenge.
• Figure 4: Query length versus total token cost for different MAS workflows on ARC-Challenge.
• Figure 5: Token cost distributions for successful and failed instances on ARC-Challenge.