Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

TS-OOD: Evaluating Time-Series Out-of-Distribution Detection and Prospective Directions for Progress

Onat Gungor, Amanda Sofie Rios, Nilesh Ahuja, Tajana Rosing

TL;DR

This work tackles the challenge of detecting out-of-distribution inputs in time-series data, where existing modality-agnostic OOD methods often underperform. It introduces TS-OOD, a comprehensive evaluation pipeline that benchmarks multiple DL backbones, time-series augmentations, and backbone-agnostic losses on semantically consistent ID/OOD splits drawn from the same dataset. The study finds that most state-of-the-art OOD methods fail to transfer well to TS domains, while approaches based on deep feature modeling (DFM) yield superior performance and efficiency, especially when paired with a contrastive MPC loss and TS augmentations. The findings provide a practical benchmark and actionable guidance for developing robust TS OOD detectors with real-world impact in manufacturing and security settings.

Abstract

Detecting out-of-distribution (OOD) data is a fundamental challenge in the deployment of machine learning models. From a security standpoint, this is particularly important because OOD test data can result in misleadingly confident yet erroneous predictions, which undermine the reliability of the deployed model. Although numerous models for OOD detection have been developed in computer vision and language, their adaptability to the time-series data domain remains limited and under-explored. Yet, time-series data is ubiquitous across manufacturing and security applications for which OOD is essential. This paper seeks to address this research gap by conducting a comprehensive analysis of modality-agnostic OOD detection algorithms. We evaluate over several multivariate time-series datasets, deep learning architectures, time-series specific data augmentations, and loss functions. Our results demonstrate that: 1) the majority of state-of-the-art OOD methods exhibit limited performance on time-series data, and 2) OOD methods based on deep feature modeling may offer greater advantages for time-series OOD detection, highlighting a promising direction for future time-series OOD detection algorithm development.

TS-OOD: Evaluating Time-Series Out-of-Distribution Detection and Prospective Directions for Progress

TL;DR

This work tackles the challenge of detecting out-of-distribution inputs in time-series data, where existing modality-agnostic OOD methods often underperform. It introduces TS-OOD, a comprehensive evaluation pipeline that benchmarks multiple DL backbones, time-series augmentations, and backbone-agnostic losses on semantically consistent ID/OOD splits drawn from the same dataset. The study finds that most state-of-the-art OOD methods fail to transfer well to TS domains, while approaches based on deep feature modeling (DFM) yield superior performance and efficiency, especially when paired with a contrastive MPC loss and TS augmentations. The findings provide a practical benchmark and actionable guidance for developing robust TS OOD detectors with real-world impact in manufacturing and security settings.

Abstract

Detecting out-of-distribution (OOD) data is a fundamental challenge in the deployment of machine learning models. From a security standpoint, this is particularly important because OOD test data can result in misleadingly confident yet erroneous predictions, which undermine the reliability of the deployed model. Although numerous models for OOD detection have been developed in computer vision and language, their adaptability to the time-series data domain remains limited and under-explored. Yet, time-series data is ubiquitous across manufacturing and security applications for which OOD is essential. This paper seeks to address this research gap by conducting a comprehensive analysis of modality-agnostic OOD detection algorithms. We evaluate over several multivariate time-series datasets, deep learning architectures, time-series specific data augmentations, and loss functions. Our results demonstrate that: 1) the majority of state-of-the-art OOD methods exhibit limited performance on time-series data, and 2) OOD methods based on deep feature modeling may offer greater advantages for time-series OOD detection, highlighting a promising direction for future time-series OOD detection algorithm development.

Paper Structure

This paper contains 19 sections, 4 figures, 3 tables.

Table of Contents

  1. Introduction
  2. TS-OOD: Evaluation Pipeline
  3. OOD detection setting
  4. In-distribution (ID) Training
  5. Deep Learning (DL) Backbones.
  6. Time-series Data Augmentation.
  7. Backbone Agnostic Loss Functions.
  8. Out-of-distribution (OOD) Detection
  9. OOD Detection Methods.
  10. OOD Scoring and Evaluation
  11. Experimental Setup
  12. Results
  13. Sensitivity to DL Backbone Architecture
  14. Impact of Augmentation Techniques
  15. Selection of the Training Loss function on post-hoc OOD detection methods
  16. ...and 4 more sections

Figures (4)

  • Figure 1: The proposed evaluation pipeline (TS-OOD) for modality-agnostic OOD detection models across a range of deep learning architectures, time-series data augmentations, and loss functions on multivariate time-series data
  • Figure 2: Impact of DL backbone on OOD detection performance across state-of-the-art deep-learning OOD detection methods
  • Figure 3: Augmentation Analysis on select datasets
  • Figure 4: Loss Function Analysis