Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

TS-Arena Technical Report -- A Pre-registered Live Forecasting Platform

Marcel Meyer, Sascha Kaltenpoth, Kevin Zalipski, Henrik Albers, Oliver Müller

TL;DR

TS-Arena tackles the TSFM evaluation crisis by turning the test set into the future, unseen at pre-registration, using live data streams and SCD2 historization to preserve provenance and prevent leakage. The platform provides a leakage-resistant, continuously updated benchmarking framework with dual participation modes (containerized and BYOP) and a moving frontier defined by $t_p$, enabling real-time, domain-spanning evaluation (starting in energy) with rolling, scope-aware leaderboards. Primary performance is tracked by $MASE$, with plans to incorporate probabilistic metrics like CRPS, and to broaden domain coverage while strengthening governance and auditability. This work demonstrates feasibility through an energy-focused prototype and outlines a scalable path toward robust, leakage-resistant forecasting benchmarks across multiple domains.

Abstract

While Time Series Foundation Models (TSFMs) offer transformative capabilities for forecasting, they simultaneously risk triggering a fundamental evaluation crisis. This crisis is driven by information leakage due to overlapping training and test sets across different models, as well as the illegitimate transfer of global patterns to test data. While the ability to learn shared temporal dynamics represents a primary strength of these models, their evaluation on historical archives often permits the exploitation of observed global shocks, which violates the independence required for valid benchmarking. We introduce TS-Arena, a platform that restores the operational integrity of forecasting by treating the genuinely unknown future as the definitive test environment. By implementing a pre-registration mechanism on live data streams, the platform ensures that evaluation targets remain physically non-existent during inference, thereby enforcing a strict global temporal split. This methodology establishes a moving temporal frontier that prevents historical contamination and provides an authentic assessment of model generalization. Initially applied within the energy sector, TS-Arena provides a sustainable infrastructure for comparing foundation models under real-world constraints. A prototype of the platform is available at https://huggingface.co/spaces/DAG-UPB/TS-Arena.

TS-Arena Technical Report -- A Pre-registered Live Forecasting Platform

TL;DR

TS-Arena tackles the TSFM evaluation crisis by turning the test set into the future, unseen at pre-registration, using live data streams and SCD2 historization to preserve provenance and prevent leakage. The platform provides a leakage-resistant, continuously updated benchmarking framework with dual participation modes (containerized and BYOP) and a moving frontier defined by , enabling real-time, domain-spanning evaluation (starting in energy) with rolling, scope-aware leaderboards. Primary performance is tracked by , with plans to incorporate probabilistic metrics like CRPS, and to broaden domain coverage while strengthening governance and auditability. This work demonstrates feasibility through an energy-focused prototype and outlines a scalable path toward robust, leakage-resistant forecasting benchmarks across multiple domains.

Abstract

While Time Series Foundation Models (TSFMs) offer transformative capabilities for forecasting, they simultaneously risk triggering a fundamental evaluation crisis. This crisis is driven by information leakage due to overlapping training and test sets across different models, as well as the illegitimate transfer of global patterns to test data. While the ability to learn shared temporal dynamics represents a primary strength of these models, their evaluation on historical archives often permits the exploitation of observed global shocks, which violates the independence required for valid benchmarking. We introduce TS-Arena, a platform that restores the operational integrity of forecasting by treating the genuinely unknown future as the definitive test environment. By implementing a pre-registration mechanism on live data streams, the platform ensures that evaluation targets remain physically non-existent during inference, thereby enforcing a strict global temporal split. This methodology establishes a moving temporal frontier that prevents historical contamination and provides an authentic assessment of model generalization. Initially applied within the energy sector, TS-Arena provides a sustainable infrastructure for comparing foundation models under real-world constraints. A prototype of the platform is available at https://huggingface.co/spaces/DAG-UPB/TS-Arena.
Paper Structure (28 sections, 1 equation, 3 figures, 1 table)

This paper contains 28 sections, 1 equation, 3 figures, 1 table.

Table of Contents

  1. Introduction
  2. Background
  3. Benchmark Concept
  4. Implementation
  5. Architecture Overview
  6. Data Ingestion and Historization
  7. APIs and Leakage Controls
  8. Challenge Orchestration
  9. Random Sampling Policy
  10. Participation Modes
  11. Evaluation and Metrics
  12. Leaderboard Design and Fairness
  13. Operational Considerations
  14. Consequences
  15. Discussion
  16. ...and 13 more sections

Figures (3)

  • Figure 1: Pre-registered forecasts of future data points.
  • Figure 2: Ranking of models based on their participated challenges in the last 7 days. On the top different filterings options are available to see model performances at specific horizons, frequencies or domains. The current results are not yet meaningful and are therefore pixelated.
  • Figure 3: Different Time Series Foundation Models predicted the future market price of Germany/Luxembourg. The actual values (punctual grey line) were collected after registration for the forecasting challenge was closed.