Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Towards Child-Inclusive Clinical Video Understanding for Autism Spectrum Disorder

Aditya Kommineni, Digbalay Bose, Tiantian Feng, So Hyun Kim, Helen Tager-Flusberg, Somer Bishop, Catherine Lord, Sudarsana Kadiri, Shrikanth Narayanan

TL;DR

This work investigates the use of foundation models across three modalities: speech, video, and text, to analyse child-focused interaction sessions and proposes a unified methodology to combine multiple modalities by using large language models as reasoning agents.

Abstract

Clinical videos in the context of Autism Spectrum Disorder are often long-form interactions between children and caregivers/clinical professionals, encompassing complex verbal and non-verbal behaviors. Objective analyses of these videos could provide clinicians and researchers with nuanced insights into the behavior of children with Autism Spectrum Disorder. Manually coding these videos is a time-consuming task and requires a high level of domain expertise. Hence, the ability to capture these interactions computationally can augment the manual effort and enable supporting the diagnostic procedure. In this work, we investigate the use of foundation models across three modalities: speech, video, and text, to analyse child-focused interaction sessions. We propose a unified methodology to combine multiple modalities by using large language models as reasoning agents. We evaluate their performance on two tasks with different information granularity: activity recognition and abnormal behavior detection. We find that the proposed multimodal pipeline provides robustness to modality-specific limitations and improves performance on the clinical video analysis compared to unimodal settings.

Towards Child-Inclusive Clinical Video Understanding for Autism Spectrum Disorder

TL;DR

This work investigates the use of foundation models across three modalities: speech, video, and text, to analyse child-focused interaction sessions and proposes a unified methodology to combine multiple modalities by using large language models as reasoning agents.

Abstract

Clinical videos in the context of Autism Spectrum Disorder are often long-form interactions between children and caregivers/clinical professionals, encompassing complex verbal and non-verbal behaviors. Objective analyses of these videos could provide clinicians and researchers with nuanced insights into the behavior of children with Autism Spectrum Disorder. Manually coding these videos is a time-consuming task and requires a high level of domain expertise. Hence, the ability to capture these interactions computationally can augment the manual effort and enable supporting the diagnostic procedure. In this work, we investigate the use of foundation models across three modalities: speech, video, and text, to analyse child-focused interaction sessions. We propose a unified methodology to combine multiple modalities by using large language models as reasoning agents. We evaluate their performance on two tasks with different information granularity: activity recognition and abnormal behavior detection. We find that the proposed multimodal pipeline provides robustness to modality-specific limitations and improves performance on the clinical video analysis compared to unimodal settings.
Paper Structure (14 sections, 2 figures, 2 tables)

This paper contains 14 sections, 2 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Datasets
  3. Remote-NLS
  4. ADOSMod3
  5. Methodology
  6. Experiments
  7. Results
  8. Activity Recognition
  9. Activity Segmentation
  10. Abnormal behaviors
  11. Multimodality vs Unimodality
  12. Context Scaling and Model Size
  13. Hallucinations
  14. Conclusion & Future Work

Figures (2)

  • Figure 1: Schematic of the proposed multimodal processing pipeline. During the modality specific content extraction, natural language descriptions of video and speech are obtained. Consequently, these descriptions are used for LLM refinement. E1, E2 and E3 are binary classification tasks. The classes for Activity Recognition and Activity Segmentation are as mentioned. Example prompts corresponding to each refinement mode are provided in the bottom table.
  • Figure 2: Class wise Activity Recognition F1 for LLaVA-NeXT-Qwen-32B