Multimodal Ensemble with Conditional Feature Fusion for Dysgraphia Diagnosis in Children from Handwriting Samples

TL;DR

This study addresses the challenge of diagnosing dysgraphia by leveraging a multimodal approach that combines online handwriting dynamics with offline image-based features. It introduces a novel dataset created by rasterizing online handwriting into offline images and proposes an ensemble with conditional feature fusion that selectively incorporates fused features when prediction confidence is uncertain. Through comprehensive experiments with SVM and XGBoost classifiers, the method achieves state-of-the-art accuracy, notably 88.8% on pseudoword data using a single word sample. The work demonstrates the practical potential of multimodal handwriting analysis for accessible, efficient dysgraphia screening, while acknowledging dataset size limitations and the need for further interpretability studies.

Abstract

Developmental dysgraphia is a neurological disorder that hinders children's writing skills. In recent years, researchers have increasingly explored machine learning methods to support the diagnosis of dysgraphia based on offline and online handwriting. In most previous studies, the two types of handwriting have been analysed separately, which does not necessarily lead to promising results. In this way, the relationship between online and offline data cannot be explored. To address this limitation, we propose a novel multimodal machine learning approach utilizing both online and offline handwriting data. We created a new dataset by transforming an existing online handwritten dataset, generating corresponding offline handwriting images. We considered only different types of word data (simple word, pseudoword & difficult word) in our multimodal analysis. We trained SVM and XGBoost classifiers separately on online and offline features as well as implemented multimodal feature fusion and soft-voted ensemble. Furthermore, we proposed a novel ensemble with conditional feature fusion method which intelligently combines predictions from online and offline classifiers, selectively incorporating feature fusion when confidence scores fall below a threshold. Our novel approach achieves an accuracy of 88.8%, outperforming SVMs for single modalities by 12-14%, existing methods by 8-9%, and traditional multimodal approaches (soft-vote ensemble and feature fusion) by 3% and 5%, respectively. Our methodology contributes to the development of accurate and efficient dysgraphia diagnosis tools, requiring only a single instance of multimodal word/pseudoword data to determine the handwriting impairment. This work highlights the potential of multimodal learning in enhancing dysgraphia diagnosis, paving the way for accessible and practical diagnostic tools.

Figures (5)

• Figure 1: Handwriting samples of word('leto'), pseudoword('lamoken'), and difficult word('hračk'arstvo') from the dataset written by the subjects in each category (typically developing and dysgraphia category.
• Figure 2: Dysgraphia diagnosis system development: work flow
• Figure 3: Overview of implemented traditional multimodal classifier fusion and feature fusion approach for dysgraphia diagnosis
• Figure 4: Overview of proposed novel multimodal classifier ensemble with conditonal feature fusion approach for dysgraphia diagnosis
• Figure 5: Effect of confidence score threshold value in multimodal ensemble with conditional feature fusion method for word and pseudo word data.