Contrastive Learning Is Not Optimal for Quasiperiodic Time Series

TL;DR

The paper addresses the mismatch between contrastive self-supervised learning objectives and the needs of quasiperiodic time series, where within-record temporal dynamics are informative but often ignored when forcing cross-record distinctiveness. It proposes Distilled Embedding for Almost-Periodic Time Series (DEAPS), a non-contrastive SSL framework that disentangles static and dynamic patterns using dual projectors, a Gradual Loss $L_{gra}$ to encourage dynamic evolution, and selective optimization with a covariance regularizer. Empirically, DEAPS yields up to about 10% improvements on downstream tasks with few labeled records across AFib identification, gender classification, and Physionet 2017, and PCA analyses illustrate clearer disentanglement of static and dynamic components compared to contrastive methods. The approach offers a practical, label-efficient pathway for physiological time series analysis and suggests broader applicability to other quasiperiodic signals, though its pretraining is currently demonstrated on a single database (SHHS).

Abstract

Despite recent advancements in Self-Supervised Learning (SSL) for time series analysis, a noticeable gap persists between the anticipated achievements and actual performance. While these methods have demonstrated formidable generalization capabilities with minimal labels in various domains, their effectiveness in distinguishing between different classes based on a limited number of annotated records is notably lacking. Our hypothesis attributes this bottleneck to the prevalent use of Contrastive Learning, a shared training objective in previous state-of-the-art (SOTA) methods. By mandating distinctiveness between representations for negative pairs drawn from separate records, this approach compels the model to encode unique record-based patterns but simultaneously neglects changes occurring across the entire record. To overcome this challenge, we introduce Distilled Embedding for Almost-Periodic Time Series (DEAPS) in this paper, offering a non-contrastive method tailored for quasiperiodic time series, such as electrocardiogram (ECG) data. By avoiding the use of negative pairs, we not only mitigate the model's blindness to temporal changes but also enable the integration of a "Gradual Loss (Lgra)" function. This function guides the model to effectively capture dynamic patterns evolving throughout the record. The outcomes are promising, as DEAPS demonstrates a notable improvement of +10% over existing SOTA methods when just a few annotated records are presented to fit a Machine Learning (ML) model based on the learned representation.

Figures (8)

• Figure 1: Evolution of model performance across training procedures for the best performing Contrastive Learning Method (blue) and the proposed method (orange). While the downstream task displayed in Figure \ref{['fig:cinc']} considers multiple annotated recordings, the one shown in Figure \ref{['fig:afib']} only considers a few of them. For easier tracking of the evolution across iterations, a polynomial has been fitted.
• Figure 2: A pair time series representing a normal and an abnormal state, belonging to the same subject are shown. In addition, another time strip that belongs to a distinct subject is displayed. The ECG morphology observed across various states within subject A exhibits consistent patterns, whereas this is different between subject A and B. These patterns are hypothesized to enable the model to categorize between two subjects while not reflecting subject state information.
• Figure 3: DEAPS illustrated. Different colors represent different temporal patterns: Green (static patterns), and purple (dynamics patterns). The different colors of the arrows indicate inputs from different recordings. While $\mathcal{L}_{sim}$ is computed between the time series representations belonging to different records, $\mathcal{L}_{gra}$ is computed using the time series belonging to the same record. The four inputs belong to the same subject. The encoder is the only component that is saved at the end of the training procedure. The other components are dismissed.
• Figure 4: A PCA has been fitted on top of the MIT-AFIB representations. The values of the different components have been studied for disentangling the static and dynamic features.
• Figure 5: Effect of incorporating different elements to the training procedure. Components inherited from other existing methods are labeled in red. The impact of adding each component is displayed by the difference in the performance of each downstream task.