Intelligence per Watt: Measuring Intelligence Efficiency of Local AI

TL;DR

This work tackles the growing bottleneck of centralized cloud LLM inference by proposing intelligence per watt (IPW) as a unified metric to evaluate local inference efficiency. It conducts a large-scale empirical study across 20+ local LMs, 8 accelerators, and 1M real-world queries to measure accuracy, energy, latency, and power, delivering a comprehensive view of local inference viability. The findings show that local LMs can accurately answer a large share of queries (88.7% overall) and that IPW has risen by 5.3x from 2023 to 2025, driven by both model innovations and hardware advances; routing queries between local and cloud models can yield substantial energy, compute, and cost reductions (60–80%) with modest accuracy tradeoffs. The paper thus demonstrates meaningful potential for redistributing LLM workloads to on-device/off-device hybrids and provides IPW as a practical benchmark, complemented by an open profiling harness to support ongoing, hardware-agnostic evaluation as the ecosystem evolves.

Abstract

Large language model (LLM) queries are predominantly processed by frontier models in centralized cloud infrastructure. Rapidly growing demand strains this paradigm, and cloud providers struggle to scale infrastructure at pace. Two advances enable us to rethink this paradigm: small LMs (<=20B active parameters) now achieve competitive performance to frontier models on many tasks, and local accelerators (e.g., Apple M4 Max) run these models at interactive latencies. This raises the question: can local inference viably redistribute demand from centralized infrastructure? Answering this requires measuring whether local LMs can accurately answer real-world queries and whether they can do so efficiently enough to be practical on power-constrained devices (i.e., laptops). We propose intelligence per watt (IPW), task accuracy divided by unit of power, as a metric for assessing capability and efficiency of local inference across model-accelerator pairs. We conduct a large-scale empirical study across 20+ state-of-the-art local LMs, 8 accelerators, and a representative subset of LLM traffic: 1M real-world single-turn chat and reasoning queries. For each query, we measure accuracy, energy, latency, and power. Our analysis reveals $3$ findings. First, local LMs can accurately answer 88.7% of single-turn chat and reasoning queries with accuracy varying by domain. Second, from 2023-2025, IPW improved 5.3x and local query coverage rose from 23.2% to 71.3%. Third, local accelerators achieve at least 1.4x lower IPW than cloud accelerators running identical models, revealing significant headroom for optimization. These findings demonstrate that local inference can meaningfully redistribute demand from centralized infrastructure, with IPW serving as the critical metric for tracking this transition. We release our IPW profiling harness for systematic intelligence-per-watt benchmarking.

Figures (13)

• Figure 1: Intelligence per Watt: A Study of Local Intelligence Efficiency. We present the first systematic study of local AI inference efficiency across models, hardware, and real-world workloads. (Left)Intelligence efficiency is defined as task accuracy per unit of power, capturing both capabilities delivered and energy consumed. (Left-Middle) We conduct comprehensive performance profiling across 20+ state-of-the-art local LMs ($\leq20B$ active parameters), diverse hardware accelerators (Apple, NVIDIA, AMD), multiple performance metrics, and 1M+ real-world queries spanning chat and reasoning tasks. (Right-Middle)Local LM capabilities are improving rapidly: win/tie rate versus frontier models increases from $23.2\%$ (2023) to $71.3\%$ (2025)---a $3.1\times$ improvement in accuracy, demonstrating that local models can accurately handle significant portions of single-turn chat and reasoning queries. (Right)Intelligence per watt improves $5.3\times$ from 2023--2025, driven by advances in both model architectures and hardware accelerators, with local accelerators showing $1.5\times$ efficiency headroom compared to enterprise-grade systems.
• Figure 2: Local Models Rival Cloud Models Across Diverse Benchmarks: Individual model performance scales with size, ranging from 31.5--69.4% for IBM Granite4-H-Small, 30.0--83.6% for Gemma3-12B, 51.5--80.4% for GPT-OSS-120B, and 66.5--89.5% for Gemini 2.5 Pro. Local routing (best local LM per query) achieves 97.8%, 88.3%, 77.0%, and 92.4% on Wildchat, NaturalReasoning, SuperGPQA, and MMLU Pro respectively, surpassing cloud routing (100%, 82.9%, 66.5%, 87.4%) on three of four benchmarks.
• Figure 3: Rapid Improvement of Local LMs across Chat and Reasoning Queries: We evaluate the performance of SOTA local models released between April 2024 and August 2025 on Wildchat and NaturalReasoning. On Wildchat (left), local models show a win/tie rate of 78.2% against Qwen3-235B as of August 2025, compared to just 28.0% in April 2024---a 2.8$\times$ improvement in 16 months. On NaturalReasoning (right), local models achieve 80.9% accuracy by August 2025, up from 48.7% in April 2024---a 66% relative improvement.
• Figure 4: Increasing GPU Memory of Consumer Accelerators: Memory capacity (GB) for local accelerators. Over the past decade, local hardware has significantly closed the memory gap with cloud-grade accelerators, particularly since 2020, driven by advances in high bandwidth memory (HBM) components and unified memory architectures.
• Figure 5: Increase in Intelligence per Joule for Local LMs and Accelerators: Efficiency improved $18.0\times$ over 16 months, decomposed into $3.1\times$ from better local LMs and $5.9\times$ from better local accelerators.