Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Energy Efficient Software Hardware CoDesign for Machine Learning: From TinyML to Large Language Models

Mohammad Saleh Vahdatpour, Yanqing Zhang

Abstract

The rapid deployment of machine learning across platforms from milliwatt-class TinyML devices to large language models has made energy efficiency a primary constraint for sustainable AI. Across these scales, performance and energy are increasingly limited by data movement and memory-system behavior rather than by arithmetic throughput alone. This work reviews energy efficient software hardware codesign methods spanning edge inference and training to datacenter-scale LLM serving, covering accelerator architectures (e.g., ASIC/FPGA dataflows, processing-/compute-in-memory designs) and system-level techniques (e.g., partitioning, quantization, scheduling, and runtime adaptation). We distill common design levers and trade-offs, and highlight recurring gaps including limited cross-platform generalization, large and costly co-design search spaces, and inconsistent benchmarking across workloads and deployment settings. Finally, we outline a hierarchical decomposition perspective that maps optimization strategies to computational roles and supports incremental adaptation, offering practical guidance for building energy and carbon aware ML systems.

Energy Efficient Software Hardware CoDesign for Machine Learning: From TinyML to Large Language Models

Abstract

The rapid deployment of machine learning across platforms from milliwatt-class TinyML devices to large language models has made energy efficiency a primary constraint for sustainable AI. Across these scales, performance and energy are increasingly limited by data movement and memory-system behavior rather than by arithmetic throughput alone. This work reviews energy efficient software hardware codesign methods spanning edge inference and training to datacenter-scale LLM serving, covering accelerator architectures (e.g., ASIC/FPGA dataflows, processing-/compute-in-memory designs) and system-level techniques (e.g., partitioning, quantization, scheduling, and runtime adaptation). We distill common design levers and trade-offs, and highlight recurring gaps including limited cross-platform generalization, large and costly co-design search spaces, and inconsistent benchmarking across workloads and deployment settings. Finally, we outline a hierarchical decomposition perspective that maps optimization strategies to computational roles and supports incremental adaptation, offering practical guidance for building energy and carbon aware ML systems.
Paper Structure (19 sections, 1 figure, 1 table)

This paper contains 19 sections, 1 figure, 1 table.

Table of Contents

  1. INTRODUCTION
  2. BACKGROUND
  3. The Memory Wall and Data Movement Bottleneck
  4. Scale-Dependent Efficiency Challenges
  5. SOFTWARE-HARDWARE CO-DESIGN
  6. Co-Design Methodologies
  7. Cross-Layer Optimization Techniques
  8. ENERGY-EFFICIENT CO-DESIGN ACROSS THE ML SPECTRUM
  9. TinyML: Ultra-Low-Power Edge Intelligence
  10. Mid-Scale: Edge-Cloud Split Computing
  11. Large-Scale: Transformers and LLMs
  12. Cross-Scale Analysis and Challenges
  13. Cross-Scale Comparison
  14. Identified Gaps and Challenges
  15. A HIERARCHICAL PERSPECTIVE ON GREEN CO-DESIGN
  16. ...and 4 more sections

Figures (1)

  • Figure 1: Timeline of representative software–hardware co-design approaches for energy-efficient machine learning (2016–2025), categorized by architectural paradigm.