Revisiting Training Scale: An Empirical Study of Token Count, Power Consumption, and Parameter Efficiency

TL;DR

This study extends neural scaling research by integrating direct energy measurements into token-scale efficiency analysis under fixed hardware and model conditions. Using a within-subject design with three token counts (500K, 1M, 2M) and the TinyLlama 1.1B model, it defines an energy-aware parameter efficiency metric $PE$ that combines $invPPL$, benchmark TFLOPS, token exposure, model size, and RMS power $RMS(W)$. Results reveal a strictly monotonic decline in parameter efficiency as token count increases, driven primarily by rises in sustained energy consumption rather than large performance differences, with robust statistical significance ($p<.001$) across token conditions. The findings emphasize the importance of energy-conscious evaluation in scaling large language models and suggest that token-scale increases may be energetically inefficient under fixed hardware and hyperparameters, impacting practical deployment and sustainability.

Abstract

Research in machine learning has questioned whether increases in training token counts reliably produce proportional performance gains in large language models. Building on prior work introducing an energy-aware parameter efficiency metric, this study empirically examines the effects of increasing training token counts under fixed hardware and training conditions. The significance of this work lies in the explicit integration of power consumption and execution duration, as reflected by the power sampling frequency, into token-scale analysis. This addresses a gap in prior studies emphasizing performance outcomes while underrepresenting computational and energy costs. Using a repeated-measures experimental design on a constant GPU instance with an identical model architecture, optimizer settings, and epoch counts, a 1.1-billion-parameter TinyLlama model was trained at three token counts (500K, 1M, and 2M). While conventional performance metrics exhibited inconsistent or diminishing returns across token scales, the inclusion of power consumption and execution duration revealed a strictly monotonic decline in training efficiency as token count increased. Repeated-measures ANOVA demonstrated a strong effect of token count on parameter efficiency, with all pairwise comparisons remaining significant following Bonferroni correction. These findings indicate that increases in training token counts may be energetically inefficient even when marginal performance improvements are observed, underscoring the importance of efficiency-aware evaluation in large language model training.