Hidden Leaks in Time Series Forecasting: How Data Leakage Affects LSTM Evaluation Across Configurations and Validation Strategies
Salma Albelali, Moataz Ahmed
TL;DR
The paper addresses data leakage in time-series forecasting and how validation design mediates leakage effects on LSTM evaluation. It introduces RMSE Gain as a metric to quantify leakage-induced distortion and a configuration-aware validation framework that treats the validation pipeline as a testable component. Through experiments on a Climate dataset across 2-way, 3-way, and 10-fold cross-validation, the study shows 10-fold CV is highly leakage-prone (RMSE Gain up to 20.5%), while 2-way/3-way splits are more robust; smaller input windows and longer lags exacerbate leakage, whereas larger windows mitigate it. The findings guide the design of leakage-resistant evaluation pipelines and emphasize reporting RMSE Gain alongside traditional metrics for trustworthy forecasting assessments.
Abstract
Deep learning models, particularly Long Short-Term Memory (LSTM) networks, are widely used in time series forecasting due to their ability to capture complex temporal dependencies. However, evaluation integrity is often compromised by data leakage, a methodological flaw in which input-output sequences are constructed before dataset partitioning, allowing future information to unintentionally influence training. This study investigates the impact of data leakage on performance, focusing on how validation design mediates leakage sensitivity. Three widely used validation techniques (2-way split, 3-way split, and 10-fold cross-validation) are evaluated under both leaky (pre-split sequence generation) and clean conditions, with the latter mitigating leakage risk by enforcing temporal separation during data splitting prior to sequence construction. The effect of leakage is assessed using RMSE Gain, which measures the relative increase in RMSE caused by leakage, computed as the percentage difference between leaky and clean setups. Empirical results show that 10-fold cross-validation exhibits RMSE Gain values of up to 20.5% at extended lag steps. In contrast, 2-way and 3-way splits demonstrate greater robustness, typically maintaining RMSE Gain below 5% across diverse configurations. Moreover, input window size and lag step significantly influence leakage sensitivity: smaller windows and longer lags increase the risk of leakage, whereas larger windows help reduce it. These findings underscore the need for configuration-aware, leakage-resistant evaluation pipelines to ensure reliable performance estimation.