Cherry-Picking in Time Series Forecasting: How to Select Datasets to Make Your Model Shine

TL;DR

Time series forecasting evaluations can be biased by dataset cherry-picking, undermining generalizability. The authors propose a four-step evaluation framework to quantify how dataset selection affects model rankings across classical and deep learning approaches, using a diverse benchmark suite. They show that cherry-picking can inflate top-performance claims (e.g., 46% best, 77% top-3 with 4 datasets) and that increasing dataset coverage reduces misidentification risk by about 40%. The work advocates rigorous, comprehensive evaluation frameworks and reproducible benchmarks to improve robustness and real-world reliability of forecasting methods.

Abstract

The importance of time series forecasting drives continuous research and the development of new approaches to tackle this problem. Typically, these methods are introduced through empirical studies that frequently claim superior accuracy for the proposed approaches. Nevertheless, concerns are rising about the reliability and generalizability of these results due to limitations in experimental setups. This paper addresses a critical limitation: the number and representativeness of the datasets used. We investigate the impact of dataset selection bias, particularly the practice of cherry-picking datasets, on the performance evaluation of forecasting methods. Through empirical analysis with a diverse set of benchmark datasets, our findings reveal that cherry-picking datasets can significantly distort the perceived performance of methods, often exaggerating their effectiveness. Furthermore, our results demonstrate that by selectively choosing just four datasets - what most studies report - 46% of methods could be deemed best in class, and 77% could rank within the top three. Additionally, recent deep learning-based approaches show high sensitivity to dataset selection, whereas classical methods exhibit greater robustness. Finally, our results indicate that, when empirically validating forecasting algorithms on a subset of the benchmarks, increasing the number of datasets tested from 3 to 6 reduces the risk of incorrectly identifying an algorithm as the best one by approximately 40%. Our study highlights the critical need for comprehensive evaluation frameworks that more accurately reflect real-world scenarios. Adopting such frameworks will ensure the development of robust and reliable forecasting methods.

Figures (4)

• Figure 1: Rank distribution of various forecasting models across all datasets. Models are organized vertically, with green bars representing classical methods and orange bars representing deep learning models..
• Figure 2: Impact of cherry-picking on the rankings of NHITS (left), Informer (center), and TCN (right). Each subfigure (1 through 6) represents the model rankings based on cherry-picked subsets. The red bars indicate the model that we are cherry-picking for.
• Figure 3: Percentage of models that could be reported as top 1, 2, and 3 performers based on an experimental setup of 4 datasets.
• Figure 4: Breakdown of the percentages for top 1, 2, and 3 positions across different numbers of datasets.