Improving Time Series Encoding with Noise-Aware Self-Supervised Learning and an Efficient Encoder

TL;DR

This work proposes an innovative training strategy that promotes consistent representation learning, accounting for the presence of noise-prone signals in natural time series, and proposes an encoder architecture that incorporates dilated convolution within the Inception block, resulting in a scalable and robust network with a wide receptive field.

Abstract

In this work, we investigate the time series representation learning problem using self-supervised techniques. Contrastive learning is well-known in this area as it is a powerful method for extracting information from the series and generating task-appropriate representations. Despite its proficiency in capturing time series characteristics, these techniques often overlook a critical factor - the inherent noise in this type of data, a consideration usually emphasized in general time series analysis. Moreover, there is a notable absence of attention to developing efficient yet lightweight encoder architectures, with an undue focus on delivering contrastive losses. Our work address these gaps by proposing an innovative training strategy that promotes consistent representation learning, accounting for the presence of noise-prone signals in natural time series. Furthermore, we propose an encoder architecture that incorporates dilated convolution within the Inception block, resulting in a scalable and robust network with a wide receptive field. Experimental findings underscore the effectiveness of our method, consistently outperforming state-of-the-art approaches across various tasks, including forecasting, classification, and abnormality detection. Notably, our method attains the top rank in over two-thirds of the classification UCR datasets, utilizing only 40% of the parameters compared to the second-best approach. Our source code for CoInception framework is accessible at https://github.com/anhduy0911/CoInception.

Figures (15)

• Figure 1: Overview of the proposed CoInception framework. We use samples from Representation Pool in different downstream tasks.
• Figure 2: Output representations for toy time series containing high-frequency noise. CoInception's results can still capture the periodic characteristics regardless of the presence of noise.
• Figure 3: Illustration of the Inception block and the accumulated receptive field upon stacking.
• Figure 4: Critical Difference Diagram comparing different classifiers on 125 Datasets from UCR Repository with the confidence level of $95\%$.
• Figure 5: Alignment analysis. Distribution of $l_2$ distance between features of positive pairs.