Scaling Laws for Energy Efficiency of Local LLMs

TL;DR

The study analyzes CPU-only local inference for LLMs and VLMs on two edge hardware regimes (MacBook Pro M2 and Raspberry Pi 5) to understand scalability under energy and compute constraints. Using a unified AUC-based profiling framework, it uncovers two scaling laws: LLM compute increases roughly linearly with token length, while VLM compute exhibits a preprocessing induced knee governed by a resize clamp. It further demonstrates that CompactifAI compression reduces CPU/RAM usage and energy, often while preserving or improving semantic accuracy, with larger gains on constrained hardware. These findings yield practical guidance for edge deployments: explicitly manage tokens and pixels, deploy compressed models by default, and document preprocessing configurations as part of system design for sustainable local inference.

Abstract

Deploying local large language models and vision-language models on edge devices requires balancing accuracy with constrained computational and energy budgets. Although graphics processors dominate modern artificial-intelligence deployment, most consumer hardware--including laptops, desktops, industrial controllers, and embedded systems--relies on central processing units. Despite this, the computational laws governing central-processing-unit-only inference for local language and vision-language workloads remain largely unexplored. We systematically benchmark large language and vision-language models on two representative central-processing-unit tiers widely used for local inference: a MacBook Pro M2, reflecting mainstream laptop-class deployment, and a Raspberry Pi 5, representing constrained, low-power embedded settings. Using a unified methodology based on continuous sampling of processor and memory usage together with area-under-curve integration, we characterize how computational load scales with input text length for language models and with image resolution for vision-language models. We uncover two empirical scaling laws: (1) computational cost for language-model inference scales approximately linearly with token length; and (2) vision-language models exhibit a preprocessing-driven "resolution knee", where compute remains constant above an internal resolution clamp and decreases sharply below it. Beyond these laws, we show that quantum-inspired compression reduces processor and memory usage by up to 71.9% and energy consumption by up to 62%, while preserving or improving semantic accuracy. These results provide a systematic quantification of multimodal central-processing-unit-only scaling for local language and vision-language workloads, and they identify model compression and input-resolution preprocessing as effective, low-cost levers for sustainable edge inference.

Figures (8)

• Figure 1: LLM CPU AUC vs. input token length for LLaMA-3.1 and Gilda v3. Panel (a) shows MacBook Pro M2 results and panel (b) shows Raspberry Pi 5 results. Gilda reduces CPU AUC by 31.3% and RAM AUC by 55.9% on the MacBook Pro M2, and by 60.5% (CPU) and 71.9% (RAM) on the Raspberry Pi 5 at matched token lengths, while also increasing throughput (2.1$\times$ on M2 and 2.6$\times$ on RPi5).
• Figure 2: VLM CPU AUC across image resolutions for Qwen2 and Axolotl. Panel (c) shows MacBook Pro M2 results and panel (d) shows Raspberry Pi 5 results. Both devices exhibit a preprocessing-induced "resolution knee": above the internal clamp, the two models have nearly identical AUC because the effective visual resolution is the same, while below the clamp compute drops sharply as the number of effective pixels decreases. Axolotl provides a modest average CPU AUC reduction (23.3% on the MacBook Pro M2 and $\approx$0.2% on the Raspberry Pi 5); the main effect illustrated here is the knee shape rather than the absolute gap between models.
• Figure 3: Throughput (tokens/s) for Axolotl vs. Qwen2 across image resolutions. Panel (a) shows MacBook Pro M2 results and panel (b) shows Raspberry Pi 5 results. Although the curves partially overlap at high resolutions, Axolotl achieves on average 1.8$\times$ higher throughput on the MacBook Pro M2 and 2.0$\times$ higher throughput on the Raspberry Pi 5, and is faster on 19/20 and 18/20 resolutions, respectively.
• Figure 4: Semantic accuracy of Axolotl and Qwen2 across image resolutions, measured as SimCSE similarity to Gemini 2.5 Flash reference answers. Panel (a) reports MacBook Pro M2 results and panel (b) reports Raspberry Pi 5 results. The curves appear visually close because of the narrow y-axis range, but Axolotl achieves higher mean accuracy on both devices: on the MacBook Pro M2, average accuracy is 77.4% for Axolotl vs. 70.5% for Qwen2 (+6.9%), and on the Raspberry Pi 5 it is 76.7% vs. 70.9% (+5.8%).
• Figure 5: Semantic accuracy of Gilda v3 and LLaMA-3.1 across the 19 LLM prompts, measured as SimCSE similarity to Gemini 2.5 Flash references. Panel (a) shows MacBook Pro M2 results and panel (b) shows Raspberry Pi 5 results. On the MacBook Pro M2, Gilda attains an average accuracy of 89.1% compared to 80.0% for LLaMA-3.1 (+9.1%). On the Raspberry Pi 5, the averages are 90.8% vs. 77.0% (+13.8%).