Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Can Generic LLMs Help Analyze Child-adult Interactions Involving Children with Autism in Clinical Observation?

Tiantian Feng, Anfeng Xu, Rimita Lahiri, Helen Tager-Flusberg, So Hyun Kim, Somer Bishop, Catherine Lord, Shrikanth Narayanan

TL;DR

This work evaluates generic LLMs' ability to analyze child-adult dyadic interactions in a clinically relevant context involving children with ASD, and shows their potential to segment interactions of interest, assist in language skills evaluation, identify engaged activities, and offer clinical-relevant context for assessments.

Abstract

Large Language Models (LLMs) have shown significant potential in understanding human communication and interaction. However, their performance in the domain of child-inclusive interactions, including in clinical settings, remains less explored. In this work, we evaluate generic LLMs' ability to analyze child-adult dyadic interactions in a clinically relevant context involving children with ASD. Specifically, we explore LLMs in performing four tasks: classifying child-adult utterances, predicting engaged activities, recognizing language skills and understanding traits that are clinically relevant. Our evaluation shows that generic LLMs are highly capable of analyzing long and complex conversations in clinical observation sessions, often surpassing the performance of non-expert human evaluators. The results show their potential to segment interactions of interest, assist in language skills evaluation, identify engaged activities, and offer clinical-relevant context for assessments.

Can Generic LLMs Help Analyze Child-adult Interactions Involving Children with Autism in Clinical Observation?

TL;DR

This work evaluates generic LLMs' ability to analyze child-adult dyadic interactions in a clinically relevant context involving children with ASD, and shows their potential to segment interactions of interest, assist in language skills evaluation, identify engaged activities, and offer clinical-relevant context for assessments.

Abstract

Large Language Models (LLMs) have shown significant potential in understanding human communication and interaction. However, their performance in the domain of child-inclusive interactions, including in clinical settings, remains less explored. In this work, we evaluate generic LLMs' ability to analyze child-adult dyadic interactions in a clinically relevant context involving children with ASD. Specifically, we explore LLMs in performing four tasks: classifying child-adult utterances, predicting engaged activities, recognizing language skills and understanding traits that are clinically relevant. Our evaluation shows that generic LLMs are highly capable of analyzing long and complex conversations in clinical observation sessions, often surpassing the performance of non-expert human evaluators. The results show their potential to segment interactions of interest, assist in language skills evaluation, identify engaged activities, and offer clinical-relevant context for assessments.

Paper Structure

This paper contains 16 sections, 5 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Datasets
  3. Task Designs
  4. Experiments
  5. Evaluation Methods
  6. Computing Infrastructure
  7. Experiment Details and Metrics
  8. Findings
  9. Utterance-level Understanding
  10. Session-level Understanding
  11. How To Improve Predictions of LLMs?
  12. Concerns with LLMs
  13. Can LLMs Outperform Non-experts?
  14. Feedback from Non-expert Evaluations
  15. Conclusion
  16. ...and 1 more sections

Figures (5)

  • Figure 1: An illustration of a part of the child-adult conversation with the protocols in Remote-NLS. All examples in this paper are scripted instead of raw due to IRB restrictions. Blue, red, and purple denote repetitions, language skills, and engaged activity, respectively. The figures on the right show a prompt template for selecting the activities that are engaged from the transcripts.
  • Figure 2: The figure shows a prompt template for recognizing language skills in both Remote-NLS and ADOSMod3 datasets.
  • Figure 3: The figure shows a prompt template for recognizing activities in both Remote-NLS and ADOSMod3 datasets.
  • Figure 4: The figure shows a prompt template for classifying age ranges in both Remote-NLS and ADOSMod3 datasets.
  • Figure 5: Exemplar output including confusion and hallucinations from the LLaMa2-7B in predicting language skills.