Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Benchmarking the Energy Savings with Speculative Decoding Strategies

Rohit Dutta, Paramita Koley, Soham Poddar, Janardan Misra, Sanjay Podder, Naveen Balani, Saptarshi Ghosh, Niloy Ganguly

TL;DR

The paper investigates whether speculative decoding (SD) strategies reduce energy use in large language models, beyond merely lowering latency. It surveys multiple SD methods (CoGA, DyGA, Eagle-2, Eagle-3) across diverse model families and benchmark tasks, quantifying energy with Code Carbon and reporting per-1K-token metrics. Key findings show that lower walltime does not always translate to energy savings; energy reductions depend on the target–assistant model gap, dataset, and decoding strategy, with Eagle variants often delivering the strongest gains while others are inconsistent. The work highlights the need for energy-aware SD design and backend considerations, emphasizing that sustainability gains are contingent on architecture, data characteristics, and implementation choices.

Abstract

Speculative decoding has emerged as an effective method to reduce latency and inference cost of LLM inferences. However, there has been inadequate attention towards the energy requirements of these models. To address this gap, this paper presents a comprehensive survey of energy requirements of speculative decoding strategies, with detailed analysis on how various factors -- model size and family, speculative decoding strategies, and dataset characteristics -- influence the energy optimizations.

Benchmarking the Energy Savings with Speculative Decoding Strategies

TL;DR

The paper investigates whether speculative decoding (SD) strategies reduce energy use in large language models, beyond merely lowering latency. It surveys multiple SD methods (CoGA, DyGA, Eagle-2, Eagle-3) across diverse model families and benchmark tasks, quantifying energy with Code Carbon and reporting per-1K-token metrics. Key findings show that lower walltime does not always translate to energy savings; energy reductions depend on the target–assistant model gap, dataset, and decoding strategy, with Eagle variants often delivering the strongest gains while others are inconsistent. The work highlights the need for energy-aware SD design and backend considerations, emphasizing that sustainability gains are contingent on architecture, data characteristics, and implementation choices.

Abstract

Speculative decoding has emerged as an effective method to reduce latency and inference cost of LLM inferences. However, there has been inadequate attention towards the energy requirements of these models. To address this gap, this paper presents a comprehensive survey of energy requirements of speculative decoding strategies, with detailed analysis on how various factors -- model size and family, speculative decoding strategies, and dataset characteristics -- influence the energy optimizations.
Paper Structure (14 sections, 3 figures, 7 tables)

This paper contains 14 sections, 3 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Experimental Setup
  3. Energy Consumption of SD Strategies
  4. Analysis of Speedup and Energy Savings
  5. Effect of Individual Model Runtime
  6. Trends across Implementation Platforms
  7. Concluding Discussions
  8. Sample prompts
  9. Model configurations
  10. Speculative Decoding strategies
  11. Hardware and Energy metrics
  12. Additional metrics
  13. Extended Results of const-X
  14. Acknowledgements

Figures (3)

  • Figure 1: Speculative decoding approaches (CoGA, DyGA) with Vicuna-13B as target ends up in consuming more energy than vanilla decoding (applying only target model) despite exhibiting lower inference latency.
  • Figure 2: Speedup ($\gamma_t$) and total energy savings ($\gamma_e^{\text{Total}}$) for speculative decoding methods across model pairs on Human-Eval and GSM-8k. Eagle consistently achieves the highest speedup and energy efficiency. Llama-based pairs under both CoGA and DyGA exhibit positive runtime and energy gains, while Vicuna-based pairs frequently underperform, with values often below unity.
  • Figure 3: Comparison of speedup ($\gamma_t$) and energy savings ($\gamma_e^{Total}$) achieved by speculative decoding across different implementational platforms -- HuggingFace and vLLM evaluated on the GSM-8k and Human-Eval. HuggingFace consistently outperforms vLLM in both metrics across almost all configurations.