Aggregate Representation Measure for Predictive Model Reusability
Vishwesh Sangarya, Richard Bradford, Jung-Eun Kim
TL;DR
The paper tackles the high resource cost of adapting pretrained neural networks to distribution shifts by introducing Aggregate Representation Measure (ARM), a metric that quantifies changes in layer-wise activations via a single forward pass. ARM aggregates per-layer shifts using the Wasserstein distance, defined as $ARM = \frac{1}{L} \sum_{l=1}^{L} WD(P_{l,d1}, P_{l,d2})$, where $P_l$ are layer distributions for original ($d1$) and shifted ($d2$) data. Empirically, ARM correlates with retraining epochs, energy consumption, and carbon emissions across multiple datasets and architectures, enabling both intra-model and inter-model comparisons to guide sustainable deployment. This work supports proactive budgeting for retraining and provides a quantitative tool for selecting reusable architectures with lower resource footprints.
Abstract
In this paper, we propose a predictive quantifier to estimate the retraining cost of a trained model in distribution shifts. The proposed Aggregated Representation Measure (ARM) quantifies the change in the model's representation from the old to new data distribution. It provides, before actually retraining the model, a single concise index of resources - epochs, energy, and carbon emissions - required for the retraining. This enables reuse of a model with a much lower cost than training a new model from scratch. The experimental results indicate that ARM reasonably predicts retraining costs for varying noise intensities and enables comparisons among multiple model architectures to determine the most cost-effective and sustainable option.