An Optimization Method for Autoregressive Time Series Forecasting

TL;DR

This work tackles the challenge of long-horizon time-series forecasting by enforcing temporal causality through an AR rollout-aware training objective. It introduces an RL-style loss that rewards monotonic growth of AR rollout errors $|x_t- ilde{x}_t|$ with horizon and concatenates short-horizon AR predictions to form long-term forecasts, all without changing model architecture. The method is model-agnostic and yields >10% MSE improvements over strong baselines like iTransformer across multiple datasets, while enabling short-horizon models to produce reliable forecasts far beyond their trained horizon. The study also reveals that AR rollout error accumulation tends to increase with rollout length and discusses theoretical bounds, practical implications, and avenues for future refinement via hybrid RL techniques and richer causal loss designs.

Abstract

Current time-series forecasting models are primarily based on transformer-style neural networks. These models achieve long-term forecasting mainly by scaling up the model size rather than through genuinely autoregressive (AR) rollout. From the perspective of large language model training, the traditional training process for time-series forecasting models ignores temporal causality. In this paper, we propose a novel training method for time-series forecasting that enforces two key properties: (1) AR prediction errors should increase with the forecasting horizon. Any violation of this principle is considered random guessing and is explicitly penalized in the loss function, and (2) the method enables models to concatenate short-term AR predictions for forming flexible long-term forecasts. Empirical results demonstrate that our method establishes a new state-of-the-art across multiple benchmarks, achieving an MSE reduction of more than 10% compared to iTransformer and other recent strong baselines. Furthermore, it enables short-horizon forecasting models to perform reliable long-term predictions at horizons over 7.5 times longer. Code is available at https://github.com/LizhengMathAi/AROpt

Figures (2)

• Figure 1: Specialized long-horizon forecasting models (vanilla training) vs. single short-horizon forecasting model (trained with our method and inferred in AR rollout mode).iTransformer (Weather), iInformer (Traffic), iFlowformer (Electricity). Our method outperforms vanilla training on short horizons. Enable small models to output arbitrary-length forecasts via AR rollouts without retraining, and surpass larger, specialized models on long-horizon forecasting.
• Figure 2: Direct long-term forecasting vs. AR rollout forecasting (a) A short-horizon forecasting model that predicts the next forecasting horizon using an encoder–projector architecture. This architecture is based solely on iTransformer, and the output window is strictly separated from the input window. In contrast, for Informer, Flashformer, and Flowformer, the projector is contained within the decoder, and they employ overlapping input–output windows. (b) A long-horizon forecasting model that reuses the same encoder but scales up the decoder/projector to directly predict longer future horizons. (c) Problem: direct long-horizon forecasting ignores temporal causality, which can lead to random guessing -- early predictions exhibit larger errors, whereas later predictions do not accumulate errors and instead produce smaller errors, potentially due to chance. (d) We propose a new loss function that enforces temporal causality: short-term AR rollout predictions should exhibit increasing error $\mathcal{L}_t$ as the time step $t$ increases. Violations of this constraint indicate random guessing and are penalized in the total loss. These short-term AR rollout predictions are then concatenated to form long-horizon predictions.