Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

OckBench: Measuring the Efficiency of LLM Reasoning

Zheng Du, Hao Kang, Song Han, Tushar Krishna, Ligeng Zhu

TL;DR

The paper addresses the gap in LLM evaluation where efficiency of reasoning, measured by decoding token count, is overlooked in favor of accuracy. It introduces OckBench, a model- and hardware-agnostic benchmark that jointly measures accuracy and token consumption in math and coding tasks, enabling analysis of accuracy–efficiency trade-offs across diverse models. The authors formalize decoding token count as an intrinsic efficiency metric and demonstrate Pareto-frontier style comparisons across 16 models, revealing substantial token-efficiency differences even among models with similar accuracy. The results underscore token efficiency as a practical concern for latency, energy use, and cost in real deployments, and the work provides a reproducible platform to drive the design of more token-efficient reasoning systems. Future directions include dynamic reasoning budgets, early-exit strategies, and expanded benchmarks to broader real-world workloads.

Abstract

Large language models such as GPT-4, Claude 3, and the Gemini series have improved automated reasoning and code generation. However, existing benchmarks mainly focus on accuracy and output quality, and they ignore an important factor: decoding token efficiency. In real systems, generating 10,000 tokens versus 100,000 tokens leads to large differences in latency, cost, and energy. In this work, we introduce OckBench, a model-agnostic and hardware-agnostic benchmark that evaluates both accuracy and token count for reasoning and coding tasks. Through experiments comparing multiple open- and closed-source models, we uncover that many models with comparable accuracy differ wildly in token consumption, revealing that efficiency variance is a neglected but significant axis of differentiation. We further demonstrate Pareto frontiers over the accuracy-efficiency plane and argue for an evaluation paradigm shift: we should no longer treat tokens as "free" to multiply. OckBench provides a unified platform for measuring, comparing, and guiding research in token-efficient reasoning. Our benchmarks are available at https://ockbench.github.io/ .

OckBench: Measuring the Efficiency of LLM Reasoning

TL;DR

The paper addresses the gap in LLM evaluation where efficiency of reasoning, measured by decoding token count, is overlooked in favor of accuracy. It introduces OckBench, a model- and hardware-agnostic benchmark that jointly measures accuracy and token consumption in math and coding tasks, enabling analysis of accuracy–efficiency trade-offs across diverse models. The authors formalize decoding token count as an intrinsic efficiency metric and demonstrate Pareto-frontier style comparisons across 16 models, revealing substantial token-efficiency differences even among models with similar accuracy. The results underscore token efficiency as a practical concern for latency, energy use, and cost in real deployments, and the work provides a reproducible platform to drive the design of more token-efficient reasoning systems. Future directions include dynamic reasoning budgets, early-exit strategies, and expanded benchmarks to broader real-world workloads.

Abstract

Large language models such as GPT-4, Claude 3, and the Gemini series have improved automated reasoning and code generation. However, existing benchmarks mainly focus on accuracy and output quality, and they ignore an important factor: decoding token efficiency. In real systems, generating 10,000 tokens versus 100,000 tokens leads to large differences in latency, cost, and energy. In this work, we introduce OckBench, a model-agnostic and hardware-agnostic benchmark that evaluates both accuracy and token count for reasoning and coding tasks. Through experiments comparing multiple open- and closed-source models, we uncover that many models with comparable accuracy differ wildly in token consumption, revealing that efficiency variance is a neglected but significant axis of differentiation. We further demonstrate Pareto frontiers over the accuracy-efficiency plane and argue for an evaluation paradigm shift: we should no longer treat tokens as "free" to multiply. OckBench provides a unified platform for measuring, comparing, and guiding research in token-efficient reasoning. Our benchmarks are available at https://ockbench.github.io/ .

Paper Structure

This paper contains 12 sections, 2 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Toward a Unified Model-Agnostic Reasoning Efficiency Framework
  3. Practical Cost of LLMs
  4. Invisible Inefficiency in Current Optimization and Evaluation
  5. OckBench Benchmark
  6. Benchmark Composition
  7. Question Combination.
  8. Experiments
  9. Setup
  10. Models and Tasks
  11. Main Result
  12. Conclusion

Figures (2)

  • Figure 1: (\ref{['fig:opensource']}) shows that even similar-sized models can exhibit a 10.7$\times$ difference in reasoning time due to varying decoding token counts. (\ref{['fig:closesource']}) shows that frontier closed-source models have comparable accuracy but vary significantly in reasoning efficiency.
  • Figure 2: Reasoning Efficiency Comparison Among 16 Models.