Xihe: Scalable Zero-Shot Time Series Learner Via Hierarchical Interleaved Block Attention

TL;DR

Xihe introduces Hierarchical Interleaved Block Attention (HIBA) to capture multi-scale temporal dependencies in time series, addressing zero-shot transfer across diverse domains. Built as a scalable TSFM family (9.5M–1.5B params), Xihe combines patch-based tokenization with intra- and inter-block attention to model local and global patterns, and uses multi-head quantile prediction to handle uncertainty across horizons. Pretrained on a 325B-time-point corpus with data-quality–aware mixing, Xihe achieves state-of-the-art zero-shot performance on GIFT-Eval, with especially strong results for the largest model and competitive efficiency for smaller variants. The work demonstrates consistent scaling behavior, strong cross-domain generalization, and practical inference efficiency, suggesting substantial impact for deployment of time-series foundation models in varied, data-scarce settings.

Abstract

The rapid advancement of time series foundation models (TSFMs) has been propelled by migrating architectures from language models. While existing TSFMs demonstrate impressive performance, their direct adoption of cross-domain architectures constrains effective capture of multiscale temporal dependencies inherent to time series data. This limitation becomes particularly pronounced during zero-shot transfer across datasets with divergent underlying patterns and sampling strategies. To address these challenges, we propose Hierarchical Interleaved Block Attention (HIBA) which employs hierarchical inter- and intra-block sparse attention to effectively capture multi-scale dependencies. Intra-block attention facilitates local information exchange, and inter-block attention operates across blocks to capture global temporal pattern interaction and dynamic evolution. Leveraging the HIBA architecture, we introduce Xihe, a scalable TSFM family spanning from an ultra-efficient 9.5M parameter configuration to high-capacity 1.5B variant. Evaluated on the comprehensive GIFT-Eval benchmark, our most compact Xihe-tiny model (9.5M) surpasses the majority of contemporary TSFMs, demonstrating remarkable parameter efficiency. More impressively, Xihe-max (1.5B) establishes new state-of-the-art zero-shot performance, surpassing previous best results by a substantial margin. This consistent performance excellence across the entire parameter spectrum provides compelling evidence for the exceptional generalization capabilities and architectural superiority of HIBA.

Figures (12)

• Figure 1: (a): The GIFT-Eval performance and parameter sizes of Xihe and existing TSFMs. Xihe achieves comparable, if not better, performance with less parameters. (b): Multi-scale dependencies in time series are prevalent and exhibit domain-specific characteristics. Effectively capturing these dependencies is essential for TSFMs to achieve optimal zero-shot performance. Left: The us_birth data is shown at different scales from top to bottom, highlighting the global trend, annual patterns, and local weekly patterns. Right: The scale of temporal dependencies differ as the domain and sampling strategies change across different series.
• Figure 2: The Xihe architecture for time series forecasting. The time series are first patched and tokenized to embedding, then processed by our HIBA module. A multi-head prediction module is utilized to produce final forecasting. The core of our method is the Multi-Scale Attention Module, detailed on the right, which hierarchically captures temporal patterns. It comprises two components: Inter-block attention models long-range, global dependencies by performing attention across entire blocks of tokens; Intra-block attention captures local patterns by applying self-attention only within each token block.
• Figure 3: Results for GIFT-Eval benchmark. Aggregated probabilistic metrics CRPS (Left Panel) and point metrics MASE (Middle Panel) scores (Lower is better) of the overall benchmark and short-, medium- and long-term CRPS (Right Panel) performances (Top 8). “TestData Leakage” denotes models that have been partially trained on the benchmark datasets. “Pretrained” indicates that the benchmark training datasets were included in the model’s training corpus, but without direct data leakage from the test set. “Zero-Shot” refers to models whose pre-training data contained neither the benchmark training set nor the test set.
• Figure 4: (a) Throughput comparison between Xihe family and other zero-shot models, where higher values indicate greater efficiency. For each sample, the look-back window length is set to the maximum supported by the compared models, and the prediction horizon is fixed at 720. (b) Zero-shot scaling characteristics of Xihe across different model sizes on the GIFT-Eval benchmark. The left panel illustrates the scaled CRPS as a function of model size, while the right panel presents the scaled MASE against model size. Each panel includes five data points corresponding to checkpoints ranging from 9.5M to 1.5B parameters.
• Figure 5: Two examples of forecasts comparison from GIFT-Eval benchmark. For each sample, we provide both the full context and Xihe-max prediction, as well as the zoomed-in prediction of other zero-shot models.