Optimization Strategies for Enhancing Resource Efficiency in Transformers & Large Language Models
Tom Wallace, Naser Ezzati-Jivan, Beatrice Ombuki-Berman
TL;DR
This work tackles the substantial resource demands of Transformer-based LLMs by systematically evaluating compression techniques. It contrasts standalone methods (quantization, distillation, pruning) with advanced hybrids (e.g., Minitron, ShearedLlama, MiniLLM) and introduces a flexible optimization equation to balance perplexity, time, and energy. Key findings show 4-bit quantization delivers strong energy savings with only modest perplexity increases, while hybrid approaches can drastically reduce model size with minimal accuracy loss (e.g., Llama-3.1-Minitron, MN-Minitron). The results guide sustainable deployment of LLMs across diverse hardware by clarifying when to prioritize energy efficiency, latency, or a balance of both, and point to future work on refining the framework and incorporating training costs and environmental considerations.
Abstract
Advancements in Natural Language Processing are heavily reliant on the Transformer architecture, whose improvements come at substantial resource costs due to ever-growing model sizes. This study explores optimization techniques, including Quantization, Knowledge Distillation, and Pruning, focusing on energy and computational efficiency while retaining performance. Among standalone methods, 4-bit Quantization significantly reduces energy use with minimal accuracy loss. Hybrid approaches, like NVIDIA's Minitron approach combining KD and Structured Pruning, further demonstrate promising trade-offs between size reduction and accuracy retention. A novel optimization equation is introduced, offering a flexible framework for comparing various methods. Through the investigation of these compression methods, we provide valuable insights for developing more sustainable and efficient LLMs, shining a light on the often-ignored concern of energy efficiency.