Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Automated Contrastive Learning Strategy Search for Time Series

Baoyu Jing, Yansen Wang, Guoxin Sui, Jing Hong, Jingrui He, Yuqing Yang, Dongsheng Li, Kan Ren

TL;DR

This paper constructs a principled search space of size over 3 × 1012, covering data augmentation, embedding transformation, contrastive pair construction, and contrastive losses, and introduces an efficient reinforcement learning algorithm, which optimizes CLS from the performance on the validation tasks, to obtain effective CLS within the space.

Abstract

In recent years, Contrastive Learning (CL) has become a predominant representation learning paradigm for time series. Most existing methods manually build specific CL Strategies (CLS) by human heuristics for certain datasets and tasks. However, manually developing CLS usually requires excessive prior knowledge about the data, and massive experiments to determine the detailed CL configurations. In this paper, we present an Automated Machine Learning (AutoML) practice at Microsoft, which automatically learns CLS for time series datasets and tasks, namely Automated Contrastive Learning (AutoCL). We first construct a principled search space of size over $3\times10^{12}$, covering data augmentation, embedding transformation, contrastive pair construction, and contrastive losses. Further, we introduce an efficient reinforcement learning algorithm, which optimizes CLS from the performance on the validation tasks, to obtain effective CLS within the space. Experimental results on various real-world datasets demonstrate that AutoCL could automatically find the suitable CLS for the given dataset and task. From the candidate CLS found by AutoCL on several public datasets/tasks, we compose a transferable Generally Good Strategy (GGS), which has a strong performance for other datasets. We also provide empirical analysis as a guide for the future design of CLS.

Automated Contrastive Learning Strategy Search for Time Series

TL;DR

This paper constructs a principled search space of size over 3 × 1012, covering data augmentation, embedding transformation, contrastive pair construction, and contrastive losses, and introduces an efficient reinforcement learning algorithm, which optimizes CLS from the performance on the validation tasks, to obtain effective CLS within the space.

Abstract

In recent years, Contrastive Learning (CL) has become a predominant representation learning paradigm for time series. Most existing methods manually build specific CL Strategies (CLS) by human heuristics for certain datasets and tasks. However, manually developing CLS usually requires excessive prior knowledge about the data, and massive experiments to determine the detailed CL configurations. In this paper, we present an Automated Machine Learning (AutoML) practice at Microsoft, which automatically learns CLS for time series datasets and tasks, namely Automated Contrastive Learning (AutoCL). We first construct a principled search space of size over , covering data augmentation, embedding transformation, contrastive pair construction, and contrastive losses. Further, we introduce an efficient reinforcement learning algorithm, which optimizes CLS from the performance on the validation tasks, to obtain effective CLS within the space. Experimental results on various real-world datasets demonstrate that AutoCL could automatically find the suitable CLS for the given dataset and task. From the candidate CLS found by AutoCL on several public datasets/tasks, we compose a transferable Generally Good Strategy (GGS), which has a strong performance for other datasets. We also provide empirical analysis as a guide for the future design of CLS.
Paper Structure (25 sections, 3 equations, 5 figures, 8 tables)

This paper contains 25 sections, 3 equations, 5 figures, 8 tables.

Table of Contents

  1. Introduction
  2. Methodology
  3. Problem Formulation
  4. Solution Space
  5. Search Algorithm
  6. Experiments
  7. Experimental Setup
  8. Direct Application of AutoCL
  9. Transferability Study
  10. Ablation Study
  11. Empirical Analysis of the Candidates
  12. Experiments in Deployed Application
  13. Conclusion and Future Work
  14. Data Augmentations
  15. Resizing (length)
  16. ...and 10 more sections

Figures (5)

  • Figure 1: Illustration of Contrastive Learning Strategy (CLS).
  • Figure 2: Illustration of the search algorithm of AutoCL.
  • Figure 3: Controller network $f_C$.
  • Figure 4: Training curves of the models.
  • Figure 5: Violin plots of 6 sub-dimensions for HAR/Yahoo/ETTh1. X-axis: options. Y-axis: ACC/F1/MSE(48 horizon). ACC/F1: the higher the better. MSE: the lower the better. Low, Medium and High correspond to $0.1\sim0.3$, $0.4\sim0.6$ and $0.7\sim0.95$.