TripCast: Pre-training of Masked 2D Transformers for Trip Time Series Forecasting

TL;DR

A novel modelling paradigm is proposed, TripCast, which treats trip time series as 2D data and learns representations through masking and reconstruction processes and notably outperforms other state-of-the-art baselines in in-domain forecasting scenarios and demonstrates strong scalability and transferability in out-domain forecasting scenarios.

Abstract

Deep learning and pre-trained models have shown great success in time series forecasting. However, in the tourism industry, time series data often exhibit a leading time property, presenting a 2D structure. This introduces unique challenges for forecasting in this sector. In this study, we propose a novel modelling paradigm, TripCast, which treats trip time series as 2D data and learns representations through masking and reconstruction processes. Pre-trained on large-scale real-world data, TripCast notably outperforms other state-of-the-art baselines in in-domain forecasting scenarios and demonstrates strong scalability and transferability in out-domain forecasting scenarios.

Figures (5)

• Figure 1: An illustration of flight booking time series data (left). The vertical axis represents the flight takeoff date, and the horizontal axis represents the booking process. Within each takeoff date (right), the booking process is shown as a 1D time series and the entire data is shown as a 2D matrix. Across different takeoff dates, the unobserved booking process is shown as a triangle.
• Figure 2: Illustration of trip time series data (a) and trip time series forecasting problem (b).
• Figure 3: Hierarchical granularities of trip time series. They can be categorized into three levels of granularities: domain, collection, and time series. Each domain contains multiple collections, and each collection contains multiple time series.
• Figure 4: The architecture of proposed TripCast model. Trip time series and covariates are stacked along the event time and leading time axes. The input data is normalized and projected to higher dimension. Then, token-level masking is applied to the projected input data. The masked input data is patched and fed into multiple transformer layers to learn predictive representations. Finally, output of the transformer layers is projected and reconstructed to estimate the unobserved values of future time steps.
• Figure 5: Accuracy versus the number of iterations during pre-training for different model sizes.