Sustainability via LLM Right-sizing
Jennifer Haase, Finn Klessascheck, Jan Mendling, Sebastian Pokutta
TL;DR
The paper addresses how organizations can deploy LLMs sustainably by moving beyond purely performance-driven benchmarks to task- and context-aware sufficiency. It introduces a scalable, two-LMM evaluation framework that tests 11 LLMs across 10 office tasks, using automated execution and triad evaluator scoring across 10 criteria, while tracking cost and energy proxies. Key findings show GPT-4o achieves the highest quality but at a high environmental and financial cost, whereas open-weight, locally deployable models like Gemma-3 and Phi-4 deliver reliable, usable results with substantially lower costs and improved data sovereignty. The work provides practical guidance for task-aware model selection and advocates a shift toward context-driven benchmarking to support responsible, sustainable LLM deployment in organizations.
Abstract
Large language models (LLMs) have become increasingly embedded in organizational workflows. This has raised concerns over their energy consumption, financial costs, and data sovereignty. While performance benchmarks often celebrate cutting-edge models, real-world deployment decisions require a broader perspective: when is a smaller, locally deployable model "good enough"? This study offers an empirical answer by evaluating eleven proprietary and open-weight LLMs across ten everyday occupational tasks, including summarizing texts, generating schedules, and drafting emails and proposals. Using a dual-LLM-based evaluation framework, we automated task execution and standardized evaluation across ten criteria related to output quality, factual accuracy, and ethical responsibility. Results show that GPT-4o delivers consistently superior performance but at a significantly higher cost and environmental footprint. Notably, smaller models like Gemma-3 and Phi-4 achieved strong and reliable results on most tasks, suggesting their viability in contexts requiring cost-efficiency, local deployment, or privacy. A cluster analysis revealed three model groups -- premium all-rounders, competent generalists, and limited but safe performers -- highlighting trade-offs between quality, control, and sustainability. Significantly, task type influenced model effectiveness: conceptual tasks challenged most models, while aggregation and transformation tasks yielded better performances. We argue for a shift from performance-maximizing benchmarks to task- and context-aware sufficiency assessments that better reflect organizational priorities. Our approach contributes a scalable method to evaluate AI models through a sustainability lens and offers actionable guidance for responsible LLM deployment in practice.