Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

RESQUE: Quantifying Estimator to Task and Distribution Shift for Sustainable Model Reusability

Vishwesh Sangarya, Jung-Eun Kim

TL;DR

RESQUE introduces a forward-looking estimator to quantify retraining costs when reusing models under distribution shifts or task changes. By separately modeling distribution shifts (RESQUEdist) and new-task adaptation (RESQUEtask) through representation-space analyses, it demonstrates strong correlations with energy consumption, carbon emissions, epochs, gradient norms, and parameter changes across CNNs and Vision Transformers. The approach leverages per-class embedding summaries, angular distances, clustering-based task separation, and ARI-based metrics to forecast retraining effort before computation. Empirical results across multiple datasets and architectures support its utility for sustainability-guided decisions in model reuse. The work provides practical tooling and design principles for greener AI deployment and reuse scenarios.

Abstract

As a strategy for sustainability of deep learning, reusing an existing model by retraining it rather than training a new model from scratch is critical. In this paper, we propose REpresentation Shift QUantifying Estimator (RESQUE), a predictive quantifier to estimate the retraining cost of a model to distributional shifts or change of tasks. It provides a single concise index for an estimate of resources required for retraining the model. Through extensive experiments, we show that RESQUE has a strong correlation with various retraining measures. Our results validate that RESQUE is an effective indicator in terms of epochs, gradient norms, changes of parameter magnitude, energy, and carbon emissions. These measures align well with RESQUE for new tasks, multiple noise types, and varying noise intensities. As a result, RESQUE enables users to make informed decisions for retraining to different tasks/distribution shifts and determine the most cost-effective and sustainable option, allowing for the reuse of a model with a much smaller footprint in the environment. The code for this work is available here: https://github.com/JEKimLab/AAAI2025RESQUE

RESQUE: Quantifying Estimator to Task and Distribution Shift for Sustainable Model Reusability

TL;DR

RESQUE introduces a forward-looking estimator to quantify retraining costs when reusing models under distribution shifts or task changes. By separately modeling distribution shifts (RESQUEdist) and new-task adaptation (RESQUEtask) through representation-space analyses, it demonstrates strong correlations with energy consumption, carbon emissions, epochs, gradient norms, and parameter changes across CNNs and Vision Transformers. The approach leverages per-class embedding summaries, angular distances, clustering-based task separation, and ARI-based metrics to forecast retraining effort before computation. Empirical results across multiple datasets and architectures support its utility for sustainability-guided decisions in model reuse. The work provides practical tooling and design principles for greener AI deployment and reuse scenarios.

Abstract

As a strategy for sustainability of deep learning, reusing an existing model by retraining it rather than training a new model from scratch is critical. In this paper, we propose REpresentation Shift QUantifying Estimator (RESQUE), a predictive quantifier to estimate the retraining cost of a model to distributional shifts or change of tasks. It provides a single concise index for an estimate of resources required for retraining the model. Through extensive experiments, we show that RESQUE has a strong correlation with various retraining measures. Our results validate that RESQUE is an effective indicator in terms of epochs, gradient norms, changes of parameter magnitude, energy, and carbon emissions. These measures align well with RESQUE for new tasks, multiple noise types, and varying noise intensities. As a result, RESQUE enables users to make informed decisions for retraining to different tasks/distribution shifts and determine the most cost-effective and sustainable option, allowing for the reuse of a model with a much smaller footprint in the environment. The code for this work is available here: https://github.com/JEKimLab/AAAI2025RESQUE

Paper Structure

This paper contains 25 sections, 5 equations, 5 figures, 2 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. RESQUE
  4. Distribution Shift
  5. Normalized Embedding Vectors for Distributions
  6. RESQUEdist from Normalized Embeddings
  7. Change of Task
  8. Quantifying Class Separation from Known Task Data to New Task Data
  9. Clustering Mechanism for the New Unknown Task
  10. Adjusted Rand Index to Quantify Class Separation
  11. Retraining Measures
  12. Carbon Emissions
  13. Energy Consumption
  14. Epochs
  15. Total Gradient Norm
  16. ...and 10 more sections

Figures (5)

  • Figure 1: Retraining vs. Training from scratch on the SVHN dataset with Gaussian noise using VGG16
  • Figure 2: RESQUE and retraining measures for different models and datasets
  • Figure 3: As for the new target task, CIFAR10, comparisons of retraining from Food101 vs. training from scratch. Retraining consumes significantly less resources, epochs, energy, and carbon, than training from scratch.
  • Figure 4: RESQUE vs. resource measures for ResNet and ViT retrained to different new target tasks. A positive relation between RESQUE and resource measures is exhibited.
  • Figure 5: CIFAR10 model trained to different target tasks. Error bars represent deviations. RESQUE has a strong linear relation with the peak performance a model can achieve on new tasks.