Denoising-Aware Contrastive Learning for Noisy Time Series

TL;DR

The paper tackles the challenge of noisy time-series data limiting self-supervised learning (SSL) performance. It introduces DEnoising-aware Contrastive Learning (DECL), an end-to-end framework that entwines denoising with contrastive SSL via a denoiser-driven objective and an automatic per-sample denoiser selection mechanism, guided by an auto-regressive Transformer-based encoder. DECL jointly optimizes an auto-regressive loss and a denoiser-weighted contrastive loss, where per-sample denoisers are selected using reconstruction errors and regularization to avoid overfitting. Across five noisy datasets, DECL yields consistent improvements in linear and fine-tuned evaluations, demonstrates robustness to varying noise levels, and offers transferable representations, with code released publicly.

Abstract

Time series self-supervised learning (SSL) aims to exploit unlabeled data for pre-training to mitigate the reliance on labels. Despite the great success in recent years, there is limited discussion on the potential noise in the time series, which can severely impair the performance of existing SSL methods. To mitigate the noise, the de facto strategy is to apply conventional denoising methods before model training. However, this pre-processing approach may not fully eliminate the effect of noise in SSL for two reasons: (i) the diverse types of noise in time series make it difficult to automatically determine suitable denoising methods; (ii) noise can be amplified after mapping raw data into latent space. In this paper, we propose denoising-aware contrastive learning (DECL), which uses contrastive learning objectives to mitigate the noise in the representation and automatically selects suitable denoising methods for every sample. Extensive experiments on various datasets verify the effectiveness of our method. The code is open-sourced.

Figures (12)

• Figure 1: A motivating analysis (more details provided in Appendix). (a-c) show that the SSL methods achieve higher performance after pre-processing the noisy time series in the PTB-XL dataset with a suitable denoising method LOESS, while the performance improvement is not obvious when processed by unsuitable methods like median filter. (d) suggests SSL methods tend to amplify the noise in the representation.
• Figure 2: Overview of the method DECL. It involves (i) auto-regressive learning, which maps raw data into latent space and exploits the representations for SSL; (ii) denoiser-driven contrastive learning, which leverages denoising method $\phi_j$ to build positive sample $\mathbf{z}_{i, j}^{(d)}$, amplifies the corresponding noise to build negative sample $\mathbf{z}_{i, j}^{(n)}$, and mitigates noise in representation learning; (iii) automatic denoiser selection, which injects Gaussian noise to data to avoid overfitting and determines suitable denoising methods for the contrastive learning.
• Figure 3: Performance comparison for semi-supervised representation learning with different percentages of labeled data.
• Figure 4: Unsupervised representation learning performance under varying degrees of data noise.
• Figure 5: Hyper-parameter analysis results.