SHARDeg: A Benchmark for Skeletal Human Action Recognition in Degraded Scenarios

TL;DR

This work tackles the problem of SHAR robustness to degraded time-series data encountered in real-world edge deployments by introducing the first degradation benchmark on NTU-RGB+D-120. It systematically evaluates five SHAR models under three degradation modes—uniform subsampling, random subsampling, and single block dropout—and assesses mitigation via simple frame interpolation. Key findings show that degradation form substantially affects accuracy, with gaps up to $>40\%$ at the same frame-rate, and that temporal regularity drives performance differences; interpolation mitigations can boost accuracy by large margins. Importantly, Rough Path Theory-based LogSigRNN demonstrates degradation-resilient behavior, achieving SoTA in most low-frame-rate cases, suggesting new avenues for robust SHAR models and degradation-aware training strategies that are practical for edge deployments.

Abstract

Computer vision (CV) models for detection, prediction or classification tasks operate on video data-streams that are often degraded in the real world, due to deployment in real-time or on resource-constrained hardware. It is therefore critical that these models are robust to degraded data, but state of the art (SoTA) models are often insufficiently assessed with these real-world constraints in mind. This is exemplified by Skeletal Human Action Recognition (SHAR), which is critical in many CV pipelines operating in real-time and at the edge, but robustness to degraded data has previously only been shallowly and inconsistently assessed. Here we address this issue for SHAR by providing an important first data degradation benchmark on the most detailed and largest 3D open dataset, NTU-RGB+D-120, and assess the robustness of five leading SHAR models to three forms of degradation that represent real-world issues. We demonstrate the need for this benchmark by showing that the form of degradation, which has not previously been considered, has a large impact on model accuracy; at the same effective frame rate, model accuracy can vary by >40% depending on degradation type. We also identify that temporal regularity of frames in degraded SHAR data is likely a major driver of differences in model performance, and harness this to improve performance of existing models by up to >40%, through employing a simple mitigation approach based on interpolation. Finally, we highlight how our benchmark has helped identify an important degradation-resistant SHAR model based in Rough Path Theory; the LogSigRNN SHAR model outperforms the SoTA DeGCN model in five out of six cases at low frame rates by an average accuracy of 6%, despite trailing the SoTA model by 11-12% on un-degraded data at high frame rates (30 FPS).

Figures (3)

• Figure 1: Test accuracy versus effective dropout rate for three different experiments, Uniform Subsampling ($a, b, c$), Random Subsampling ($d, e, f$) and Single block Dropout ($g, h, i$) and five SHAR models (see key in panel $h$). The columns show experiments where removed frames are not replaced (left), where linear interpolation is used to mitigate degradation at inference time by replacing missing frames (central), and the difference (right). Panel $a$ also shows equivalent frame-rates in FPS for translation from effective dropout rate. Black circles in $a, d, g$ highlight DeGCN performance at $90\%$ effective dropout rate across all three experiments, and black rectangles in $b, e, h$ show data points that are presented in Table \ref{['Tab:Baseline']}; all data are provided in Supplementary Tables 1-6.
• Figure 2: Comparison of LogSigRNN (blue triangles) and DeGCN (green squares) models with mitigation through post-hoc linear interpolation, for random subsampling (dashed lines), uniform subsampling (solid lines) and single block dropout (dotted lines). Results are shown for both cross subject (left) and cross setup (right) train-test splits.
• Figure 3: Test accuracy versus effective dropout rate under the cross-setup train-test split for three different experiments, Uniform Subsampling ($a, b, c$), Random Subsampling ($d, e, f$) and Single block Dropout ($g, h, i$) and five SHAR models (see key in panel $h$). The columns show experiments where removed frames are not replaced (left), where linear interpolation is used to mitigate degradation at inference time by replacing missing frames (central), and the difference (right). Panel $a$ also shows equivalent frame-rates in FPS for translation from effective dropout rate. All data are provided in Supplementary Tables 8-13.