Compute-Accuracy Pareto Frontiers for Open-Source Reasoning Large Language Models

TL;DR

This work introduces a compute-aware evaluation framework that maps the Pareto frontier between inference cost and reasoning accuracy for open-source LLMs across five benchmarks. By decomposing FLOPs into attention, FFN, and, crucially, MoE and Mamba components, the authors show that Mixture-of-Experts models offer superior accuracy-per-FLOP and can sustain longer reasoning traces within fixed budgets. They also reveal a trace-length asymmetry where incorrect reasoning consumes more compute than correct reasoning, and identify task-dependent knees where additional compute yields diminishing returns, highlighting the limits of brute-force scaling. Temporal analysis indicates progress in efficiency, with newer models expanding the Pareto frontier, but the study emphasizes real-world deployment considerations such as latency, energy, and cost. Overall, the findings guide practitioners toward sparsity-enabled architectures and smarter compute budgeting for robust, efficient reasoning systems.

Abstract

Large Language Models (LLMs) are demonstrating rapid improvements on complex reasoning benchmarks, particularly when allowed to utilize intermediate reasoning steps before converging on a final solution. However, current literature often overlooks the significant computational burden associated with generating long reasoning sequences. For industrial applications, model selection depends not only on raw accuracy but also on resource constraints and inference costs. In this work, we conduct a test-time-compute aware evaluation of both contemporary and older open-source LLMs, mapping their Pareto frontiers across math- and reasoning-intensive benchmarks. Our findings identify the Mixture of Experts (MoE) architecture as a strong candidate to balance performance and efficiency in our evaluation setting. Furthermore, we trace the trajectory of Pareto efficiency over time to derive an emergent trend regarding accuracy gain per unit of compute. Finally, we demonstrate that there is a saturation point for inference-time compute. Beyond a certain threshold, accuracy gains diminish, indicating that while extended reasoning capabilities are beneficial, they cannot overcome intrinsic model limitations regarding specific complexities.

Figures (6)

• Figure 1: The Reasoning Pareto Frontier. Each point represents the efficiency of a model averaged across the five benchmarks. The $x$-axis shows the average KV-aware estimation of FLOPs per query (log scale), and the $y$-axis shows task accuracy. Color indicates model type (dense in blue, mixture of experts in yellow), and marker size is proportional to the total model parameter count.
• Figure 2: The Reasoning Pareto Frontier. Each point represents the efficiency of a model on the given benchmark. The $x$-axis denotes the average FLOP estimation per query (log scale), while the $y$-axis tracks task accuracy. Color distinguishes model architecture (Dense: blue, MoE: yellow), and marker size is proportional to parameter count. Note the task-dependent saturation points, particularly visible in the lighter GSM8K versus the steeper ascent in the reasoning heavy AIME-2025 benchmark.
• Figure 3: Temporal Progress. Efficiency score (average accuracy across all five benchmarks normalized by $\log_{10}(\text{FLOPs})$) versus model release date. The trend indicates a consistent improvement in reasoning efficiency, with increasing variance in 2025 driven by specialized reasoning models.
• Figure 4: Trace Length Asymmetry. Average compute cost of incorrect versus correct traces. Points above the red dashed line ($y=x$) indicate models that spend more compute when failing. 97% of evaluated models exhibit this inefficiency.
• Figure 5: FLOPs distribution across architectural variants at sequence length 4096. GQA reduces K/V projection costs (orange/yellow), while gated FFN (SwiGLU) increases the FFN share (purple). MoE models show dominant FFN contribution due to sparse expert activation.