XMTC: Explainable Early Classification of Multivariate Time Series in Reach-to-Grasp Hand Kinematics

TL;DR

The paper tackles early prediction of user intent in reach-to-grasp hand kinematics from non-synchronized multivariate time series and introduces XMTC, a visualization-backed explainable analytics tool. By leveraging DrCIF within a meta-ensemble, XMTC enables early, accurate predictions while providing global and per-series explanations through temporal accuracy plots, confusion matrices, heatmaps, and PDPs. Case studies on real R2G data show that meaningful predictions can be achieved well before grasping, with clear trade-offs between earliness and accuracy and insights into feature contributions. The work offers a practical, generalizable framework for latency-aware decision support in HCI and related domains, with potential extensions to healthcare, robotics, and beyond.

Abstract

Hand kinematics can be measured in Human-Computer Interaction (HCI) with the intention to predict the user's intention in a reach-to-grasp action. Using multiple hand sensors, multivariate time series data are being captured. Given a number of possible actions on a number of objects, the goal is to classify the multivariate time series data, where the class shall be predicted as early as possible. Many machine-learning methods have been developed for such classification tasks, where different approaches produce favorable solutions on different data sets. We, therefore, employ an ensemble approach that includes and weights different approaches. To provide a trustworthy classification production, we present the XMTC tool that incorporates coordinated multiple-view visualizations to analyze the predictions. Temporal accuracy plots, confusion matrix heatmaps, temporal confidence heatmaps, and partial dependence plots allow for the identification of the best trade-off between early prediction and prediction quality, the detection and analysis of challenging classification conditions, and the investigation of the prediction evolution in an overview and detail manner. We employ XMTC to real-world HCI data in multiple scenarios and show that good classification predictions can be achieved early on with our classifier as well as which conditions are easy to distinguish, which multivariate time series measurements impose challenges, and which features have most impact.

Figures (18)

• Figure 1: Weights (y-axis) of the four classifiers in HIVE-COTE2 shown for all trained models for increasing time series lengths (x-axis). The plot demonstrates that DrCIF (red) receives higher weights across all models over time when compared to the other three classifiers.
• Figure 2: Overview of the XMTC tool, showing the user interface elements of the settings panel (a, b, c) and visualizations using coordinated views (d, e). The accuracy plot (d) illustrates the performance of prediction models with increasing time window (blue), accompanied by a histogram representing the distribution of the time series lengths of the entire dataset (gray) and the test dataset (yellow). Users can select a dataset in the settings panel (a) to explore model performance over time (d). Additionally, users can analyze individual test time series by selecting them in panel (b) and observing the models' class probabilities over time (e).
• Figure 3: Overview of XMTC tool (continuation of Figure \ref{['fig:tool_overview_page1']}). Users can select a model in the setting panel(c) and observe the Partial Dependence Plot (f) to find distinguishable features and patterns.
• Figure 4: Hovering information of the accuracy plot. When the user hovers over the accuracy plot, details about the hovered model are displayed, including its confusion matrix visualization, window size, accuracy, the number of time series in the entire dataset shorter than the hovered window size, and the number of time series in the test dataset shorter than the window size.
• Figure 5: Hovering information of the heatmap: The window size and the probability predictions of the model trained on the corresponding window size are displayed and a vertical line is drawn to indicate the current time point.