Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Improving Time Series Classification with Representation Soft Label Smoothing

Hengyi Ma, Weitong Chen

TL;DR

This work tackles overfitting in time series classification by introducing representation soft label smoothing, which builds soft labels from latent representations produced by a pre-trained TS2Vec encoder and distances in latent space. The proposed method is combined with a distillation-like loss and evaluated across six models with varying complexity, showing consistent improvements over hard-label training and competitive results versus standard label smoothing and confidence penalty. Key contributions include (i) encoder-based soft-label construction, (ii) integration with LS and CP for regularization, and (iii) extensive experiments on the UCR dataset family demonstrating robustness across models and tasks. The approach offers a flexible mechanism to inject similarity information into supervision, with potential applicability beyond TSC wherever suitable encoders are available.

Abstract

Previous research has indicated that deep neural network based models for time series classification (TSC) tasks are prone to overfitting. This issue can be mitigated by employing strategies that prevent the model from becoming overly confident in its predictions, such as label smoothing and confidence penalty. Building upon the concept of label smoothing, we propose a novel approach to generate more reliable soft labels, which we refer to as representation soft label smoothing. We apply label smoothing, confidence penalty, and our method representation soft label smoothing to several TSC models and compare their performance with baseline method which only uses hard labels for training. Our results demonstrate that the use of these enhancement techniques yields competitive results compared to the baseline method. Importantly, our method demonstrates strong performance across models with varying structures and complexities.

Improving Time Series Classification with Representation Soft Label Smoothing

TL;DR

This work tackles overfitting in time series classification by introducing representation soft label smoothing, which builds soft labels from latent representations produced by a pre-trained TS2Vec encoder and distances in latent space. The proposed method is combined with a distillation-like loss and evaluated across six models with varying complexity, showing consistent improvements over hard-label training and competitive results versus standard label smoothing and confidence penalty. Key contributions include (i) encoder-based soft-label construction, (ii) integration with LS and CP for regularization, and (iii) extensive experiments on the UCR dataset family demonstrating robustness across models and tasks. The approach offers a flexible mechanism to inject similarity information into supervision, with potential applicability beyond TSC wherever suitable encoders are available.

Abstract

Previous research has indicated that deep neural network based models for time series classification (TSC) tasks are prone to overfitting. This issue can be mitigated by employing strategies that prevent the model from becoming overly confident in its predictions, such as label smoothing and confidence penalty. Building upon the concept of label smoothing, we propose a novel approach to generate more reliable soft labels, which we refer to as representation soft label smoothing. We apply label smoothing, confidence penalty, and our method representation soft label smoothing to several TSC models and compare their performance with baseline method which only uses hard labels for training. Our results demonstrate that the use of these enhancement techniques yields competitive results compared to the baseline method. Importantly, our method demonstrates strong performance across models with varying structures and complexities.
Paper Structure (23 sections, 2 theorems, 5 equations, 6 figures, 2 tables, 1 algorithm)

This paper contains 23 sections, 2 theorems, 5 equations, 6 figures, 2 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related work
  3. Time Series Classification
  4. Hard Label and Label-Related Regularization Methods
  5. Knowledge Distillation
  6. Methodology
  7. Preliminaries
  8. Label smoothing
  9. Confidence penalty
  10. Knowledge distillation
  11. Proposed Method
  12. Sample confidence and soft label construction
  13. Experiment Setup
  14. Experimental Environment
  15. Dataset
  16. ...and 8 more sections

Key Result

theorem thmcountertheorem

Assuming the correct category of sample $m_i$ is A, then $\text{argmax}\, (m_i) = A$. We must ensure that the correct category in the constructed soft label has the highest confidence, so that the sample is classified into the correct category.

Figures (6)

  • Figure 1: The structure of our method.
  • Figure 2: The use of an encoder generates different representation spaces for the samples, which are measured using L2 distance later.
  • Figure 3: Hard labels, labels produced by label smoothing and labels produced by our method.
  • Figure 4: Critical difference diagrams for 6 models with different methods
  • Figure 5: Comparison of accuracy for 128 sub-datasets between our method and baselines
  • ...and 1 more figures

Theorems & Definitions (2)

  • theorem thmcountertheorem
  • theorem thmcountertheorem