Continuous-Time Linear Positional Embedding for Irregular Time Series Forecasting

TL;DR

This work proposes CTLPE, a method learning a continuous linear function for encoding temporal information which outperforms existing techniques across various irregularly-sampled time series datasets, showcasing its enhanced efficacy.

Abstract

Irregularly sampled time series forecasting, characterized by non-uniform intervals, is prevalent in practical applications. However, previous research have been focused on regular time series forecasting, typically relying on transformer architectures. To extend transformers to handle irregular time series, we tackle the positional embedding which represents the temporal information of the data. We propose CTLPE, a method learning a continuous linear function for encoding temporal information. The two challenges of irregular time series, inconsistent observation patterns and irregular time gaps, are solved by learning a continuous-time function and concise representation of position. Additionally, the linear continuous function is empirically shown superior to other continuous functions by learning a neural controlled differential equation-based positional embedding, and theoretically supported with properties of ideal positional embedding. CTLPE outperforms existing techniques across various irregularly-sampled time series datasets, showcasing its enhanced efficacy.

Figures (3)

• Figure 1: Overall framework of continuous-time learnable positional embedding (CTLPE).
• Figure 2: Sinusoidal positional embedding's correlation between time difference and average positional embedding distance.
• Figure 3: Sinusoidal positional embedding and NCDE-PE visualized.

Theorems & Definitions (2)

• Theorem 3.1
• proof