HandyLabel: Towards Post-Processing to Real-Time Annotation Using Skeleton Based Hand Gesture Recognition

TL;DR

HandyLabel tackles the bottleneck of data annotation by enabling real-time, gesture-based labeling using skeleton-based hand pose preprocessing. Through HaGRID-based evaluation, a ResNet50 backbone with Skeleton Type 1 achieves an F1 of 0.923, supporting accurate, low-latency labeling on standard hardware. A user study with 46 participants shows 88.9% preferring HandyLabel over Label Studio, highlighting gains in intuitiveness and setup simplicity. The work demonstrates a practical, privacy-conscious, web-based tool that reduces annotation time and cognitive workload while enabling scalable, real-time behavioral annotation across diverse contexts.

Abstract

The success of machine learning is deeply linked to the availability of high-quality training data, yet retrieving and manually labeling new data remains a time-consuming and error-prone process. Traditional annotation tools, such as Label Studio, often require post-processing, where users label data after it has been recorded. Post-processing is highly time-consuming and labor-intensive, especially with large datasets, and may lead to erroneous annotations due to the difficulty of subjects' memory tasks when labeling cognitive activities such as emotions or comprehension levels. In this work, we introduce HandyLabel, a real-time annotation tool that leverages hand gesture recognition to map hand signs for labeling. The application enables users to customize gesture mappings through a web-based interface, allowing for real-time annotations. To ensure the performance of HandyLabel, we evaluate several hand gesture recognition models on an open-source hand sign (HaGRID) dataset, with and without skeleton-based preprocessing. We discovered that ResNet50 with preprocessed skeleton-based images performs an F1-score of 0.923. To validate the usability of HandyLabel, a user study was conducted with 46 participants. The results suggest that 88.9% of participants preferred HandyLabel over traditional annotation tools.

Figures (6)

• Figure 1: Overview of the HandyLabel: Existing annotation tools perform through post-processing, where the user tries to make an annotation by recalling the target label (i.e., emotions). However, recalling might cause false memory and wrong annotation. Instead, our proposed application allows participants to make annotations during the experiment, enabling real-time precision.
• Figure 2: Selected hand gestures (\ref{['fig:hand_gestures']}) and preprocessing configurations for gesture recognition (\ref{['fig:pre-processing']}).
• Figure 3: System overview. The workflow begins with real-time hand gesture capture via a camera, followed by immediate processing in the web application. Captured frames are transmitted to a backend server, where a skeleton-based ResNet50 model recognizes the gestures and generates corresponding annotations. These annotated results are instantly visualized in the web interface, enabling users to perform fast, accurate, and intuitive data labeling with minimal latency.
• Figure 4: HandyLabel workflow: (\ref{['fig:app_stage1']}) Users select one-to-one relationship of the annotation label and hand sign, (\ref{['fig:app_stage2']}) Camera recording starts and hand sign recognition will be running in the backend and when model recognize the selected hand gestures, then the timestamp and the label will be stored as a log. (\ref{['fig:app_stage3']}) Lastly, after user stop recording, the dashboard shows the datetime/timestamp and the duration of each annotation. The data can be exported as a CSV file.
• Figure 5: User preferences for HandyLabel compared to Label Studio. The first pie chart illustrates that the vast majority of users found HandyLabel more intuitive than Label Studio, with only a small percentage favoring Label Studio. The second chart reflects user preferences for data annotation, with 88.9% of participants indicating they would rather use HandyLabel for annotation tasks. The third chart shows that the majority would recommend HandyLabel, highlighting a strong overall preference for HandyLabel.