CONTRAST: Continual Multi-source Adaptation to Dynamic Distributions

TL;DR

Through theoretical analysis and experimental analysis, the proposed method is able to optimally combine the source models and prioritize updates to the model least prone to forgetting, and performance does not degrade as the test data distribution changes over time.

Abstract

Adapting to dynamic data distributions is a practical yet challenging task. One effective strategy is to use a model ensemble, which leverages the diverse expertise of different models to transfer knowledge to evolving data distributions. However, this approach faces difficulties when the dynamic test distribution is available only in small batches and without access to the original source data. To address the challenge of adapting to dynamic distributions in such practical settings, we propose Continual Multi-source Adaptation to Dynamic Distributions (CONTRAST), a novel method that optimally combines multiple source models to adapt to the dynamic test data. CONTRAST has two distinguishing features. First, it efficiently computes the optimal combination weights to combine the source models to adapt to the test data distribution continuously as a function of time. Second, it identifies which of the source model parameters to update so that only the model which is most correlated to the target data is adapted, leaving the less correlated ones untouched; this mitigates the issue of ``forgetting" the source model parameters by focusing only on the source model that exhibits the strongest correlation with the test batch distribution. Through theoretical analysis we show that the proposed method is able to optimally combine the source models and prioritize updates to the model least prone to forgetting. Experimental analysis on diverse datasets demonstrates that the combination of multiple source models does at least as well as the best source (with hindsight knowledge), and performance does not degrade as the test data distribution changes over time (robust to forgetting).

Figures (4)

• Figure 1: Problem setup. Consider several source models trained using data from different weather conditions. During the deployment of these models, they may encounter varying weather conditions that could be a combination of multiple conditions in varying proportions (represented by the pie charts on top). Our goal is to infer on the test data using the ensemble of models by automatically figuring out proper combination weights and adapting the appropriate models on the fly.
• Figure 2: Overall Framework. During test time, we aim to adapt multiple source models in a manner such that it optimally blends the sources with suitable weights based on the current test distribution. Additionally, we update the parameters of only one model that exhibits the strongest correlation with the test distribution.
• Figure 3: Comparison with baselines in terms of source knowledge forgetting. Maintaining the same setting as in Table \ref{['tab:cifar100']}, we demonstrate that by integrating single-source methods with CONTRAST, the source knowledge is better preserved during dynamic adaptation. Unlike all these single-source methods, our algorithm demonstrates virtually no forgetting throughout the entire adaptation process.
• Figure 4: Visual Comparison of CONTRAST with Baselines for Semantic Segmentation Task. Each row in the figure corresponds to a different weather condition (rain, snow, fog, and night from top to bottom). It is evident that CONTRAST outperforms the baselines in terms of segmentation results.

Theorems & Definitions (6)

• Theorem 1: Convergence of Optimization \ref{['opt:main_opt']}.
• proof
• Theorem 2
• proof
• proof : Proof of Theroem \ref{['Thm:convergence']}
• proof : Proof of Theorem \ref{['theorem:thm1']}