AdaFRUGAL: Adaptive Memory-Efficient Training with Dynamic Control
Quang-Hung Bui, Anh Son Ta
TL;DR
Training large language models is memory-bound by optimizer states; AdaFRUGAL introduces two dynamic controls to the FRUGAL framework: a linear decay of the state-full subspace ratio $\rho$ and a loss-aware adaptive schedule for the update frequency $T$, aiming to reduce memory and computation without sacrificing convergence. The method is validated across English C4 and Vietnamese VietVault pre-training and GLUE fine-tuning, showing competitive accuracy with substantial resource savings, especially in memory footprint and training time. The results demonstrate that Dynamic-$T$ yields best final perplexities and faster convergence, while Dynamic-$\rho$ delivers notable memory reductions, and the Combined variant offers a balanced trade-off. Overall, AdaFRUGAL provides an autonomous, scalable approach to resource-efficient full-parameter training of large language models, reducing manual hyperparameter tuning while maintaining strong performance.
Abstract
Training Large Language Models (LLMs) is highly memory-intensive due to optimizer state overhead. The FRUGAL framework mitigates this with gradient splitting, but its static hyperparameters -- the subspace ratio ($ρ$) and update frequency ($T$) -- require costly manual tuning, limiting adaptability. We present AdaFRUGAL, which automates this process by introducing two dynamic controls: (i) a linear decay for $ρ$ to progressively reduce memory, and (ii) a loss-aware schedule for $T$ to lower computational overhead. Experiments across large-scale pre-training (English C4, Vietnamese VietVault) and fine-tuning (GLUE) demonstrate that AdaFRUGAL achieves a compelling trade-off. It maintains competitive performance against AdamW and static FRUGAL while significantly reducing both GPU memory and training time, offering a more practical, autonomous solution for resource-constrained LLM training.