Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Rolling-Origin Validation Reverses Model Rankings in Multi-Step PM10 Forecasting: XGBoost, SARIMA, and Persistence

Federico Garcia Crespi, Eduardo Yubero Funes, Marina Alfosea Simon

Abstract

(a) Many air quality forecasting studies report gains from machine learning, but evaluations often use static chronological splits and omit persistence baselines, so the operational added value under routine updating is unclear. (b) Using 2,350 daily PM10 observations from 2017 to 2024 at an urban background monitoring station in southern Europe, we compare XGBoost and SARIMA against persistence under a static split and a rolling-origin protocol with monthly updates. We report horizon-specific skill and the predictability horizon, defined as the maximum horizon with positive persistence-relative skill. Static evaluation suggests XGBoost performs well from one to seven days ahead, but rolling-origin evaluation reverses rankings: XGBoost is not consistently better than persistence at short and intermediate horizons, whereas SARIMA remains positively skilled across the full range. (c) For researchers, static splits can overstate operational usefulness and change rankings. For practitioners, rolling-origin, persistence-referenced skill profiles show which methods stay reliable at each lead time.

Rolling-Origin Validation Reverses Model Rankings in Multi-Step PM10 Forecasting: XGBoost, SARIMA, and Persistence

Abstract

(a) Many air quality forecasting studies report gains from machine learning, but evaluations often use static chronological splits and omit persistence baselines, so the operational added value under routine updating is unclear. (b) Using 2,350 daily PM10 observations from 2017 to 2024 at an urban background monitoring station in southern Europe, we compare XGBoost and SARIMA against persistence under a static split and a rolling-origin protocol with monthly updates. We report horizon-specific skill and the predictability horizon, defined as the maximum horizon with positive persistence-relative skill. Static evaluation suggests XGBoost performs well from one to seven days ahead, but rolling-origin evaluation reverses rankings: XGBoost is not consistently better than persistence at short and intermediate horizons, whereas SARIMA remains positively skilled across the full range. (c) For researchers, static splits can overstate operational usefulness and change rankings. For practitioners, rolling-origin, persistence-referenced skill profiles show which methods stay reliable at each lead time.
Paper Structure (24 sections, 2 equations, 3 figures)

This paper contains 24 sections, 2 equations, 3 figures.

Table of Contents

  1. Introduction
  2. Methodology
  3. Study Design
  4. Study Area and Data
  5. Forecasting Protocols
  6. Forecasting Models
  7. Forecast Horizons
  8. Evaluation Metrics
  9. Definition of the Predictability Horizon, H*
  10. Comparative Logic
  11. Results
  12. Case Study Results: Elche Urban Background Station
  13. Data Characteristics
  14. Static Chronological Split: Apparent Skill Stability
  15. Rolling-Origin Evaluation: XGBoost
  16. ...and 9 more sections

Figures (3)

  • Figure 1: Two-panel diagnostic summary of the daily PM10 series used in the empirical case study. Panel (a) shows the full daily time series. Panel (b) summarizes monthly seasonality through the distribution of monthly mean and median concentrations across the annual cycle.
  • Figure 2: Static-split H$^*$ evaluation for XGBoost on the empirical case study. Under a single chronological train/test partition, XGBoost remains above persistence across all horizons ($\mathrm{SS}=0.231$--$0.299$), yielding a nominal $H^*=7$. This result is informative as a single-split benchmark, but it is not representative of the rolling-origin ranking reported in the main text.
  • Figure 3: Pseudocode for H$^*$ computation under rolling-origin validation. Key constraint: all preprocessing parameters estimated exclusively from training data at each fold (step 2a).