ML-EcoLyzer: Quantifying the Environmental Cost of Machine Learning Inference Across Frameworks and Hardware

TL;DR

The paper addresses the environmental cost of ML inference across frameworks and hardware, a gap in current benchmarking. It introduces ML-EcoLyzer, an open-source tool that measures energy, carbon, thermal, and water footprints for inference and defines the Environmental Sustainability Score (ESS) to compare models across sizes and precisions, using $ESS = \frac{M}{CO_2}$ with $M = \frac{N \cdot QF}{10^6}$. Through over 1,900 inference runs across modalities, hardware tiers, and precisions, the study shows that quantization substantially reduces carbon cost, that newer architectures can be more environmentally efficient per parameter, and that hardware-task alignment is essential for sustainability. The work provides a practical benchmark and actionable guidance for deployment decisions in resource-constrained settings, and invites community adoption and extension through its open-source framework.

Abstract

Machine learning inference occurs at a massive scale, yet its environmental impact remains poorly quantified, especially on low-resource hardware. We present ML-EcoLyzer, a cross-framework tool for measuring the carbon, energy, thermal, and water costs of inference across CPUs, consumer GPUs, and datacenter accelerators. The tool supports both classical and modern models, applying adaptive monitoring and hardware-aware evaluation. We introduce the Environmental Sustainability Score (ESS), which quantifies the number of effective parameters served per gram of CO$_2$ emitted. Our evaluation covers over 1,900 inference configurations, spanning diverse model architectures, task modalities (text, vision, audio, tabular), hardware types, and precision levels. These rigorous and reliable measurements demonstrate that quantization enhances ESS, huge accelerators can be inefficient for lightweight applications, and even small models may incur significant costs when implemented suboptimally. ML-EcoLyzer sets a standard for sustainability-conscious model selection and offers an extensive empirical evaluation of environmental costs during inference.

Figures (4)

• Figure 1: Overview of ML-EcoLyzer. The framework quantifies the environmental impact of machine learning inference by integrating various dataset modalities, diverse model architectures, and hardware platforms.
• Figure 2: Average CO$_2$ emissions per minute by model family on text generation tasks (rate-based metric). This complements Table \ref{['tab:family-results']}, which reports per-inference costs. Per-minute metrics reveal throughput efficiency during sustained operation, while per-inference metrics show task-level environmental cost. Error bars: standard deviation.
• Figure 3: Average CO$_2$ emissions per minute by model family on classification tasks. Error bars: standard deviation.
• Figure 4: Sustainability Score (ESS, MP/g CO$_2$) vs. CO$_2$ emissions per inference (kg) for all model families.