Early Classification of Time Series: A Survey and Benchmark

TL;DR

Early Classification of Time Series (ECTS) aims to trigger predictions as early as possible while controlling misclassification and delay costs. The paper presents a principle-based taxonomy of separable ECTS methods, a shared evaluation protocol using AvgCost metrics, and a large-scale benchmark across varied cost settings with an open-source library and 34 datasets. It finds that cost-informed and anticipation-based methods generally outperform cost-uninformed or myopic ones, that classifier calibration can substantially affect performance, and that z-normalization offers limited practical gains in the proposed benchmark. This work provides a reusable benchmark and codebase to advance fair, reproducible evaluation of ECTS methods and guides future research in cost-aware early prediction.

Abstract

In many situations, the measurements of a studied phenomenon are provided sequentially, and the prediction of its class needs to be made as early as possible so as not to incur too high a time penalty, but not too early and risk paying the cost of misclassification. This problem has been particularly studied in the case of time series, and is known as Early Classification of Time Series (ECTS). Although it has been the subject of a growing body of literature, there is still a lack of a systematic, shared evaluation protocol to compare the relative merits of the various existing methods. In this paper, we highlight the two components of an ECTS system: decision and prediction, and focus on the approaches that separate them. This document begins by situating these methods within a principle-based taxonomy. It defines dimensions for organizing their evaluation and then reports the results of a very extensive set of experiments along these dimensions involving nine state-of-the-art ECTS algorithms. In addition, these and other experiments can be carried out using an open-source library in which most of the existing ECTS algorithms have been implemented (see https://github.com/ML-EDM/ml_edm).

Figures (19)

• Figure 1: Experiments diagram. While this section mainly discusses results about the cost settings in Subsections \ref{['sec_exp_results_std_cost']} and \ref{['sec_anomaly_cost']}, many other alternative experiments are briefly analyzed in Subsection \ref{['sec_other_experiments']} and are more detailed in the Appendix \ref{['app_results']}. Details on the used datasets can be found in Appendix \ref{['app_data']}.
• Figure 2: The ranking plot (a) shows that, across all values of $\alpha$, a top group of four approaches distinguishes itself. The significance of this result is supported by statistical tests. Specifically, we report this for $\alpha = 0.5$ as shown in (b).
• Figure 3: Pareto front, displaying for each $\alpha$ the Accuracy on the $y$-axis and $Earliness$ on the $x$-axis. Best approaches are located on the top left corner. In zoomed boxes, on the right of the Figure, points corresponding to a single $\alpha$ are highlighted, while other points are smaller and gray. Each of the trigger model is optimizing the trade-off in its own way, resulting in many different approaches having points in the Pareto dominant set.
• Figure 4: Representative delay cost (a) and misclassification ones (b) for an anomaly detection scenario. In our experiments, $\alpha \in [0,1]$.
• Figure 5: The ranking plot (a) shows that, across all $\alpha$, a top group composed by three approaches distinguish. This result is significant as supported by statistical tests. Specifically, for $\alpha = 0.5$ as shown in (b).