Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Toward Interpretable Evaluation Measures for Time Series Segmentation

Félix Chavelli, Paul Boniol, Michaël Thomazo

TL;DR

This paper tackles the weak interpretability of time series segmentation evaluation by introducing two measures: Weighted Adjusted Rand Index (WARI), which adds boundary-aware weighting to the traditional ARI, and State Matching Score (SMS), which maps predicted states to ground-truth states and scores error blocks by type and context. WARI provides position-sensitive discrimination of segmentation errors, while SMS delivers fine-grained, customizable penalties for four fundamental error types, enhancing diagnostic insight. Through synthetic and real-world benchmarks across multiple datasets and methods, the authors show that WARI and SMS yield more informative assessments than traditional measures and reveal error provenance that was previously inaccessible. The work contributes to more interpretable, robust evaluation frameworks that can guide model selection, tuning, and ensembling in time series segmentation tasks.

Abstract

Time series segmentation is a fundamental task in analyzing temporal data across various domains, from human activity recognition to energy monitoring. While numerous state-of-the-art methods have been developed to tackle this problem, the evaluation of their performance remains critically limited. Existing measures predominantly focus on change point accuracy or rely on point-based measures such as Adjusted Rand Index (ARI), which fail to capture the quality of the detected segments, ignore the nature of errors, and offer limited interpretability. In this paper, we address these shortcomings by introducing two novel evaluation measures: WARI (Weighted Adjusted Rand Index), that accounts for the position of segmentation errors, and SMS (State Matching Score), a fine-grained measure that identifies and scores four fundamental types of segmentation errors while allowing error-specific weighting. We empirically validate WARI and SMS on synthetic and real-world benchmarks, showing that they not only provide a more accurate assessment of segmentation quality but also uncover insights, such as error provenance and type, that are inaccessible with traditional measures.

Toward Interpretable Evaluation Measures for Time Series Segmentation

TL;DR

This paper tackles the weak interpretability of time series segmentation evaluation by introducing two measures: Weighted Adjusted Rand Index (WARI), which adds boundary-aware weighting to the traditional ARI, and State Matching Score (SMS), which maps predicted states to ground-truth states and scores error blocks by type and context. WARI provides position-sensitive discrimination of segmentation errors, while SMS delivers fine-grained, customizable penalties for four fundamental error types, enhancing diagnostic insight. Through synthetic and real-world benchmarks across multiple datasets and methods, the authors show that WARI and SMS yield more informative assessments than traditional measures and reveal error provenance that was previously inaccessible. The work contributes to more interpretable, robust evaluation frameworks that can guide model selection, tuning, and ensembling in time series segmentation tasks.

Abstract

Time series segmentation is a fundamental task in analyzing temporal data across various domains, from human activity recognition to energy monitoring. While numerous state-of-the-art methods have been developed to tackle this problem, the evaluation of their performance remains critically limited. Existing measures predominantly focus on change point accuracy or rely on point-based measures such as Adjusted Rand Index (ARI), which fail to capture the quality of the detected segments, ignore the nature of errors, and offer limited interpretability. In this paper, we address these shortcomings by introducing two novel evaluation measures: WARI (Weighted Adjusted Rand Index), that accounts for the position of segmentation errors, and SMS (State Matching Score), a fine-grained measure that identifies and scores four fundamental types of segmentation errors while allowing error-specific weighting. We empirically validate WARI and SMS on synthetic and real-world benchmarks, showing that they not only provide a more accurate assessment of segmentation quality but also uncover insights, such as error provenance and type, that are inaccessible with traditional measures.
Paper Structure (21 sections, 6 equations, 13 figures, 4 tables, 2 algorithms)

This paper contains 21 sections, 6 equations, 13 figures, 4 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Background and Foundations
  3. Time Series Segmentation: A Multifaceted Problem
  4. Change Point Detection
  5. State Detection
  6. Evaluating Time Series Segmentation: Typology of Errors and Desired Properties
  7. Existing Measures and Limitations
  8. Change Point Detection Measures
  9. State Detection Measures
  10. WARI and SMS: Our Proposed Measures
  11. Toward Position-Sensitivity: The Weighted Adjusted Rand Index
  12. Weighted Contingency Matrix:
  13. Properties:
  14. Sensitivity to the position parameter $\alpha$:
  15. Enhancing Interpretability: The State Matching Score
  16. ...and 6 more sections

Figures (13)

  • Figure 1: Illustration of Change Point Detection vs. State Detection.
  • Figure 2: Ground truth (top) with four error examples below: delay, isolation, transition, and missing.
  • Figure 3: Limitations of (a) F1, (b) Covering, and (c) ARI scores. For two different segmentations (S1 more accurate than S2 according to the ground truth GT), all measures return the same score.
  • Figure 4: Synthetic data examples illustrating various error types and measure responses.
  • Figure 5: Segmentation of a time series from the MoCap dataset using E2USD and Time2State.
  • ...and 8 more figures

Theorems & Definitions (2)

  • Definition 1: Real-valued Time Series
  • Definition 2: State Sequence