Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

DIVINE: Coordinating Multimodal Disentangled Representations for Oro-Facial Neurological Disorder Assessment

Mohd Mujtaba Akhtar, Girish, Muskaan Singh

TL;DR

DIVINE tackles automated, interpretable assessment of neuro-facial disorders by fusing synchronized audio and facial video through a fully disentangled multimodal framework. It leverages hierarchical variational bottlenecks to separate modality-private from cross-modal shared information, uses cross-modal alignment and sparse gated fusion, and injects clinically meaningful symptom tokens to guide aggregation. The approach supports multitask objectives—diagnosis across Healthy Control/ALS/Stroke and five clinician-rated severity scores—and demonstrates state-of-the-art performance on the Toronto NeuroFace dataset, notably achieving 98.26% accuracy with DeepSeek-VL2 and TRILLsson under full modality. DIVINE remains robust under missing modalities and unimodal conditions, highlighting potential for clinically relevant screening and longitudinal monitoring, while acknowledging the need for cross-dataset validation and careful deployment in real-world settings with human-in-the-loop safeguards.

Abstract

In this study, we present a multimodal framework for predicting neuro-facial disorders by capturing both vocal and facial cues. We hypothesize that explicitly disentangling shared and modality-specific representations within multimodal foundation model embeddings can enhance clinical interpretability and generalization. To validate this hypothesis, we propose DIVINE a fully disentangled multimodal framework that operates on representations extracted from state-of-the-art (SOTA) audio and video foundation models, incorporating hierarchical variational bottlenecks, sparse gated fusion, and learnable symptom tokens. DIVINE operates in a multitask learning setup to jointly predict diagnostic categories (Healthy Control,ALS, Stroke) and severity levels (Mild, Moderate, Severe). The model is trained using synchronized audio and video inputs and evaluated on the Toronto NeuroFace dataset under full (audio-video) as well as single-modality (audio-only and video-only) test conditions. Our proposed approach, DIVINE achieves SOTA result, with the DeepSeek-VL2 and TRILLsson combination reaching 98.26% accuracy and 97.51% F1-score. Under modality-constrained scenarios, the framework performs well, showing strong generalization when tested with video-only or audio-only inputs. It consistently yields superior performance compared to unimodal models and baseline fusion techniques. To the best of our knowledge, DIVINE is the first framework that combines cross-modal disentanglement, adaptive fusion, and multitask learning to comprehensively assess neurological disorders using synchronized speech and facial video.

DIVINE: Coordinating Multimodal Disentangled Representations for Oro-Facial Neurological Disorder Assessment

TL;DR

DIVINE tackles automated, interpretable assessment of neuro-facial disorders by fusing synchronized audio and facial video through a fully disentangled multimodal framework. It leverages hierarchical variational bottlenecks to separate modality-private from cross-modal shared information, uses cross-modal alignment and sparse gated fusion, and injects clinically meaningful symptom tokens to guide aggregation. The approach supports multitask objectives—diagnosis across Healthy Control/ALS/Stroke and five clinician-rated severity scores—and demonstrates state-of-the-art performance on the Toronto NeuroFace dataset, notably achieving 98.26% accuracy with DeepSeek-VL2 and TRILLsson under full modality. DIVINE remains robust under missing modalities and unimodal conditions, highlighting potential for clinically relevant screening and longitudinal monitoring, while acknowledging the need for cross-dataset validation and careful deployment in real-world settings with human-in-the-loop safeguards.

Abstract

In this study, we present a multimodal framework for predicting neuro-facial disorders by capturing both vocal and facial cues. We hypothesize that explicitly disentangling shared and modality-specific representations within multimodal foundation model embeddings can enhance clinical interpretability and generalization. To validate this hypothesis, we propose DIVINE a fully disentangled multimodal framework that operates on representations extracted from state-of-the-art (SOTA) audio and video foundation models, incorporating hierarchical variational bottlenecks, sparse gated fusion, and learnable symptom tokens. DIVINE operates in a multitask learning setup to jointly predict diagnostic categories (Healthy Control,ALS, Stroke) and severity levels (Mild, Moderate, Severe). The model is trained using synchronized audio and video inputs and evaluated on the Toronto NeuroFace dataset under full (audio-video) as well as single-modality (audio-only and video-only) test conditions. Our proposed approach, DIVINE achieves SOTA result, with the DeepSeek-VL2 and TRILLsson combination reaching 98.26% accuracy and 97.51% F1-score. Under modality-constrained scenarios, the framework performs well, showing strong generalization when tested with video-only or audio-only inputs. It consistently yields superior performance compared to unimodal models and baseline fusion techniques. To the best of our knowledge, DIVINE is the first framework that combines cross-modal disentanglement, adaptive fusion, and multitask learning to comprehensively assess neurological disorders using synchronized speech and facial video.
Paper Structure (27 sections, 17 equations, 3 figures, 9 tables)

This paper contains 27 sections, 17 equations, 3 figures, 9 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Representations
  4. Modeling
  5. Local Temporal Refinement
  6. Local VAE (VAE_window)
  7. Global Average Pooling
  8. Utterance-Level VAE (VAE_utterance)
  9. Cross-Modal Alignment
  10. Sparse Gated Fusion
  11. Token Injection and Dense Layer
  12. Output Heads
  13. Joint Loss Function
  14. Experiments
  15. Ablation Study
  16. ...and 12 more sections

Figures (3)

  • Figure 1: Overview of the study setup.
  • Figure 2: Proposed modeling architecture : DIVINE
  • Figure 3: Representing DIVINE configurations. Each displays true versus predicted class distributions across the combined diagnosis and severity categories: (a) DeepSeek‐VL2+TRILLsson; (b) DeepSeek‐VL2+X-vector; (c) DeepSeek‐VL2+X-vector (testing only video); (d) DeepSeek‐VL2+TRILLsson (testing only audio); (e) ViViT (Multitask); (f) WavLM; (g) Kinematic (Multitask); and (h) Kinematic (Classification). These matrices highlight classification consistency and error patterns for each fusion pairing.