NuwaTS: a Foundation Model Mending Every Incomplete Time Series

TL;DR

NuwaTS is a novel framework that repurposes Pre-trained Language Models (PLMs) for general time series imputation, and outperforms state-of-the-art domain-specific models across various datasets under the proposed benchmarking protocol.

Abstract

Time series imputation is critical for many real-world applications and has been widely studied. However, existing models often require specialized designs tailored to specific missing patterns, variables, or domains which limits their generalizability. In addition, current evaluation frameworks primarily focus on domain-specific tasks and often rely on time-wise train/validation/test data splits, which fail to rigorously assess a model's ability to generalize across unseen variables or domains. In this paper, we present \textbf{NuwaTS}, a novel framework that repurposes Pre-trained Language Models (PLMs) for general time series imputation. Once trained, NuwaTS can be applied to impute missing data across any domain. We introduce specialized embeddings for each sub-series patch, capturing information about the patch, its missing data patterns, and its statistical characteristics. By combining contrastive learning with the imputation task, we train PLMs to create a versatile, one-for-all imputation model. Additionally, we employ a plug-and-play fine-tuning approach, enabling efficient adaptation to domain-specific tasks with minimal adjustments. To evaluate cross-variable and cross-domain generalization, we propose a new benchmarking protocol that partitions the datasets along the variable dimension. Experimental results on over seventeen million time series samples from diverse domains demonstrate that NuwaTS outperforms state-of-the-art domain-specific models across various datasets under the proposed benchmarking protocol. Furthermore, we show that NuwaTS generalizes to other time series tasks, such as forecasting. Our codes are available at https://github.com/Chengyui/NuwaTS.

Figures (15)

• Figure 1: (a) Cross-variable and cross-domain generalization: time series data across different variables and domains may exhibit both shared and distinct patterns. (b) The conventional train/validation/test division protocol of partitioning datasets along the time dimension. (c) Variable-wise division: the proposed approach trains, validates, and tests models on distinct sets of variables, ensuring the model's ability to generalize across unseen variables during deployment.
• Figure 2: Overview of NuwaTS. To fully leverage the semantic information of time series and their missing patterns, NuwaTS introduces the tokenization of time series in patches. It utilizes the missing data patterns, statistical information for each pattern and the entire series, and a domain-specific embedding, trained through imputation and contrastive learning tasks.
• Figure 2: Imputation results on four PEMS datasets with the same setting as Table \ref{['bigtable1']}.
• Figure 3: Illustration of domain-specific fine-tuning.
• Figure 4: The four different versions of NuwaTS trained in this study.

Theorems & Definitions (1)

• Definition 3.1