Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

DQS: A Low-Budget Query Strategy for Enhancing Unsupervised Data-driven Anomaly Detection Approaches

Lucas Correia, Jan-Christoph Goos, Thomas Bäck, Anna V. Kononova

TL;DR

The paper tackles threshold selection in truly unsupervised time-series anomaly detection by integrating active learning with an existing unsupervised detector. It introduces the dissimilarity-based query strategy (DQS), which uses dynamic time warping to select diverse samples for oracle labeling to optimize the anomaly-score threshold. Empirical results on the PATH dataset show DQS performs best in small-budget scenarios and remains competitive with larger budgets, though some strategies exhibit greater robustness to mislabelling; overall, active-learning-based thresholding consistently outperforms the purely unsupervised threshold. The work highlights practical guidance for selecting query strategies, emphasizes the impact of mislabelling, and calls for standardized benchmarking to advance robust active-learning approaches in discrete-sequence anomaly detection.

Abstract

Truly unsupervised approaches for time series anomaly detection are rare in the literature. Those that exist suffer from a poorly set threshold, which hampers detection performance, while others, despite claiming to be unsupervised, need to be calibrated using a labelled data subset, which is often not available in the real world. This work integrates active learning with an existing unsupervised anomaly detection method by selectively querying the labels of multivariate time series, which are then used to refine the threshold selection process. To achieve this, we introduce a novel query strategy called the dissimilarity-based query strategy (DQS). DQS aims to maximise the diversity of queried samples by evaluating the similarity between anomaly scores using dynamic time warping. We assess the detection performance of DQS in comparison to other query strategies and explore the impact of mislabelling, a topic that is underexplored in the literature. Our findings indicate that DQS performs best in small-budget scenarios, though the others appear to be more robust when faced with mislabelling. Therefore, in the real world, the choice of query strategy depends on the expertise of the oracle and the number of samples they are willing to label. Regardless, all query strategies outperform the unsupervised threshold even in the presence of mislabelling. Thus, whenever it is feasible to query an oracle, employing an active learning-based threshold is recommended.

DQS: A Low-Budget Query Strategy for Enhancing Unsupervised Data-driven Anomaly Detection Approaches

TL;DR

The paper tackles threshold selection in truly unsupervised time-series anomaly detection by integrating active learning with an existing unsupervised detector. It introduces the dissimilarity-based query strategy (DQS), which uses dynamic time warping to select diverse samples for oracle labeling to optimize the anomaly-score threshold. Empirical results on the PATH dataset show DQS performs best in small-budget scenarios and remains competitive with larger budgets, though some strategies exhibit greater robustness to mislabelling; overall, active-learning-based thresholding consistently outperforms the purely unsupervised threshold. The work highlights practical guidance for selecting query strategies, emphasizes the impact of mislabelling, and calls for standardized benchmarking to advance robust active-learning approaches in discrete-sequence anomaly detection.

Abstract

Truly unsupervised approaches for time series anomaly detection are rare in the literature. Those that exist suffer from a poorly set threshold, which hampers detection performance, while others, despite claiming to be unsupervised, need to be calibrated using a labelled data subset, which is often not available in the real world. This work integrates active learning with an existing unsupervised anomaly detection method by selectively querying the labels of multivariate time series, which are then used to refine the threshold selection process. To achieve this, we introduce a novel query strategy called the dissimilarity-based query strategy (DQS). DQS aims to maximise the diversity of queried samples by evaluating the similarity between anomaly scores using dynamic time warping. We assess the detection performance of DQS in comparison to other query strategies and explore the impact of mislabelling, a topic that is underexplored in the literature. Our findings indicate that DQS performs best in small-budget scenarios, though the others appear to be more robust when faced with mislabelling. Therefore, in the real world, the choice of query strategy depends on the expertise of the oracle and the number of samples they are willing to label. Regardless, all query strategies outperform the unsupervised threshold even in the presence of mislabelling. Thus, whenever it is feasible to query an oracle, employing an active learning-based threshold is recommended.

Paper Structure

This paper contains 13 sections, 3 equations, 3 figures, 3 tables, 4 algorithms.

Table of Contents

  1. Introduction
  2. Background
  3. Active Learning
  4. Dynamic Time Warping
  5. Problem Description
  6. Related Work
  7. Proposed Approach
  8. Experiments
  9. Impact of Mislabelling Probability
  10. Impact of Query Budgets
  11. Benchmarking Considerations
  12. Results
  13. Conclusion and Outlook

Figures (3)

  • Figure 1: Interaction between the oracle and a model-based anomaly detector.
  • Figure 2: $F_1$ scores for each query strategy plotted as a function of time. The results in tabular form are also shown in Table \ref{['tab:results_budgets']}. The grey upper bound represents the results using the hypothetical best threshold $\tau_\text{best}$, and the grey lower bound represents the results using the unsupervised threshold $\tau_\text{us}$.
  • Figure 3: $F_1$ scores for each query strategy plotted as a function of time. The results in tabular form are also shown in Table \ref{['tab:results_mislabelling']}. The grey upper bound represents the results using the hypothetical best threshold $\tau_\text{best}$, and the grey lower bound represents the results using the unsupervised threshold $\tau_\text{us}$.