Evaluating Classifier Confidence for Surface EMG Pattern Recognition
Akira Furui
TL;DR
The paper addresses the need for calibrated confidence in EMG pattern recognition by evaluating both discriminative and generative classifiers on four upper-limb EMG datasets. It analyzes posterior probabilities $p(y|\mathbf{x})$ via reliability diagrams and metrics like $ECE$ and $MCE$ to assess calibration. The key finding is that the scale mixture model–based generative classifier (SMMC) achieves both high accuracy and well-calibrated confidence (often $ECE<10\%$), while deep discriminative networks, although accurate, can be overconfident. This suggests that incorporating variance uncertainty through a generative framework can improve reliability for online EMG interfaces and adaptive learning.
Abstract
Surface electromyogram (EMG) can be employed as an interface signal for various devices and software via pattern recognition. In EMG-based pattern recognition, the classifier should not only be accurate, but also output an appropriate confidence (i.e., probability of correctness) for its prediction. If the confidence accurately reflects the likelihood of true correctness, then it will be useful in various application tasks, such as motion rejection and online adaptation. The aim of this paper is to identify the types of classifiers that provide higher accuracy and better confidence in EMG pattern recognition. We evaluate the performance of various discriminative and generative classifiers on four EMG datasets, both visually and quantitatively. The analysis results show that while a discriminative classifier based on a deep neural network exhibits high accuracy, it outputs a confidence that differs from true probabilities. By contrast, a scale mixture model-based classifier, which is a generative classifier that can account for uncertainty in EMG variance, exhibits superior performance in terms of both accuracy and confidence.