Byte Pair Encoding for Efficient Time Series Forecasting

TL;DR

The paper tackles inefficiencies in time series tokenization by proposing a motif-based tokenization that builds a discrete vocabulary of recurring temporal patterns and adaptively compresses data. It introduces a two-step tokenizer that quantizes real-valued series and encodes recurring motifs, plus a lightweight conditional decoding step to recover discretization error without gradients. Across Chronos foundation models and five datasets, the approach yields substantial improvements in forecasting accuracy and efficiency, with strong zero-shot generalization and interpretable token embeddings. This method has the potential to reduce compute and energy consumption in large time series models while maintaining or improving predictive performance.

Abstract

Existing time series tokenization methods predominantly encode a constant number of samples into individual tokens. This inflexible approach can generate excessive tokens for even simple patterns like extended constant values, resulting in substantial computational overhead. Inspired by the success of byte pair encoding, we propose the first pattern-centric tokenization scheme for time series analysis. Based on a discrete vocabulary of frequent motifs, our method merges samples with underlying patterns into tokens, compressing time series adaptively. Exploiting our finite set of motifs and the continuous properties of time series, we further introduce conditional decoding as a lightweight yet powerful post-hoc optimization method, which requires no gradient computation and adds no computational overhead. On recent time series foundation models, our motif-based tokenization improves forecasting performance by 36% and boosts efficiency by 1990% on average. Conditional decoding further reduces MSE by up to 44%. In an extensive analysis, we demonstrate the adaptiveness of our tokenization to diverse temporal patterns, its generalization to unseen data, and its meaningful token representations capturing distinct time series properties, including statistical moments and trends.

Figures (22)

• Figure 1: Motif-based tokenization transforms time series data (gray) through a two-step process: 1) quantizing samples into discrete bins, 2) merging recurring patterns of variable length into representative motifs (equivalent motifs share the same color). Motif repetition is highlighted by their x‑axis projection.
• Figure 2: (a) Our motif-based tokenization first quantizes a time series into symbols and finds recurring motifs as tokens, building a discrete vocabulary. (b) Based on the compressed motif sequence, a neural network forecasts the time series through a categorical distribution over our vocabulary. (c) Finally, we propose conditional decoding to reduce the discretization error when transforming tokens back to their continuous representation.
• Figure 3: Zero-shot comparison between our motif-based and sample-wise tokenization (Chronos) on Electricity.
• Figure 4: Conditional decoding improves forecasting quality for 3.0 tokenizers in small models on 2.0 datasets.
• Figure 5: Our adaptive tokenizer (a) exploits periodically recurring motifs on the Traffic dataset and (b-d) compresses time series adaptively depending on pattern complexity on the Weather dataset.