Estimating Environmental Cost Throughout Model's Adaptive Life Cycle

TL;DR

The paper tackles the rising environmental burden of training and retraining neural networks by introducing PreIndex, a lightweight predictive index for estimating environmental and compute costs when adapting models to distributional shifts. PreIndex fuses three components—$p$ (average representation distance), $inv\text{_}ari$ (complement of ARI), and $s$ (noise-variance scaling)—into a single score computed from a single forward pass, with the formula $PreIndex = (p + inv\text{_}ari) / (1 + (p + (1 - ari)) * s) - s\text{_}bar$. It is evaluated across CNNs and Vision Transformers on CIFAR10/100 and TinyImageNet, showing strong correlations with retraining resources such as epochs, gradient norms, and parameter changes, as well as with energy and carbon emissions tracked by CodeCarbon; results support its model-agnostic applicability and practical utility for sustainable model reuse. Overall, PreIndex enables data-driven, environmentally conscious decisions about when and how to retrain, reducing the environmental footprint of deploying adaptive AI systems.

Abstract

With the rapid increase in the research, development, and application of neural networks in the current era, there is a proportional increase in the energy needed to train and use models. Crucially, this is accompanied by the increase in carbon emissions into the environment. A sustainable and socially beneficial approach to reducing the carbon footprint and rising energy demands associated with the modern age of AI/deep learning is the adaptive and continuous reuse of models with regard to changes in the environment of model deployment or variations/changes in the input data. In this paper, we propose PreIndex, a predictive index to estimate the environmental and compute resources associated with model retraining to distributional shifts in data. PreIndex can be used to estimate environmental costs such as carbon emissions and energy usage when retraining from current data distribution to new data distribution. It also correlates with and can be used to estimate other resource indicators associated with deep learning, such as epochs, gradient norm, and magnitude of model parameter change. PreIndex requires only one forward pass of the data, following which it provides a single concise value to estimate resources associated with retraining to the new distribution shifted data. We show that PreIndex can be reliably used across various datasets, model architectures, different types, and intensities of distribution shifts. Thus, PreIndex enables users to make informed decisions for retraining to different distribution shifts and determine the most cost-effective and sustainable option, allowing for the reuse of a model with a much smaller footprint in the environment. The code for this work is available here: https://github.com/JEKimLab/AIES2024PreIndex

Figures (5)

• Figure 1: Training from scratch vs. retraining of ResNet18 on CIFAR10 with Poisson noise. Retraining consumes clearly less resources than training from scratch.
• Figure 2: PreIndex vs. resource indicators on ResNet and ViT. Results of more architectures are presented in the Appendix.
• Figure 3: PreIndex vs resource indicators for various models retrained to CIFAR10 with distribution shift
• Figure 4: PreIndex vs resource indicators for various models retrained to CIFAR100 with distribution shift
• Figure 5: PreIndex vs resource indicators for GoogleNet retrained to TinyImageNet with distribution shift