The impact of data set similarity and diversity on transfer learning success in time series forecasting

TL;DR

The paper addresses how source-target similarity and source diversity affect transfer learning for time series forecasting in zero-shot and fine-tuned regimes. It conducts a controlled study with five public source datasets and five target datasets, using DeepAR-based pre-training (including a Multisource foundation-style model) and comparing against Scratch and ETS baselines, while linking performance to feature-based (tsfresh, catch22) and shape-based (DTW) data-set characteristics. The main findings show that higher catch22 diversity improves zero-shot accuracy and uncertainty, while greater tsfresh similarity to the target reduces zero-shot bias; fine-tuning generally reduces dependence on source data and enhances performance, though some patterns vary by target. The work provides practical guidance for selecting source data and supports the case for diverse foundation-model-like pre-training, with implications for building public time-series repositories to support transfer learning.

Abstract

Pre-trained models have become pivotal in enhancing the efficiency and accuracy of time series forecasting on target data sets by leveraging transfer learning. While benchmarks validate the performance of model generalization on various target data sets, there is no structured research providing similarity and diversity measures to explain which characteristics of source and target data lead to transfer learning success. Our study pioneers in systematically evaluating the impact of source-target similarity and source diversity on zero-shot and fine-tuned forecasting outcomes in terms of accuracy, bias, and uncertainty estimation. We investigate these dynamics using pre-trained neural networks across five public source datasets, applied to forecasting five target data sets, including real-world wholesales data. We identify two feature-based similarity and diversity measures, finding that source-target similarity reduces forecasting bias, while source diversity improves forecasting accuracy and uncertainty estimation, but increases the bias.

Figures (5)

• Figure 1: Summary of our approach and research question.
• Figure 2: Zero-shot/scratch (red circles) and fine-tuned (blue circles) accuracy in AvgRMSSE (see Eq. \ref{['AvgRMSSE']}) per source model across the five target data sets. View in color.
• Figure 3: PCAs of the tsfresh and catch22 features of the source and target data sets with the latter including the Multisource one. Best viewed in color.
• Figure 4: Relation between the catch22 (a+c) and tsfresh (b) source diversity and their zero-shot transfer learning forecast accuracy error (a), bias (b) and uncertainty estimation error (c). Each curve corresponds to a target data set and each point to a source data set with its respective diversity on the x-axis and its zero-shot transfer learning forecasting performance on the y-axis.
• Figure 5: Relations between tsfresh source-target distance and zero-shot transfer learning bias. Each curve corresponds to a target data set and each point to a source data set with its respective tsfresh feature distance on the x-axis and the zero-shot transfer learning bias on the y-axis.