Video-based Analysis Reveals Atypical Social Gaze in People with Autism Spectrum Disorder

TL;DR

This study quantitatively analyzed four gaze features: gaze engagement, gaze variance, gaze density map, and gaze diversion frequency and developed a classifier trained on these features to identify gaze abnormalities in ASD participants.

Abstract

In this study, we present a quantitative and comprehensive analysis of social gaze in people with autism spectrum disorder (ASD). Diverging from traditional first-person camera perspectives based on eye-tracking technologies, this study utilizes a third-person perspective database from the Autism Diagnostic Observation Schedule, 2nd Edition (ADOS-2) interview videos, encompassing ASD participants and neurotypical individuals as a reference group. Employing computational models, we extracted and processed gaze-related features from the videos of both participants and examiners. The experimental samples were divided into three groups based on the presence of social gaze abnormalities and ASD diagnosis. This study quantitatively analyzed four gaze features: gaze engagement, gaze variance, gaze density map, and gaze diversion frequency. Furthermore, we developed a classifier trained on these features to identify gaze abnormalities in ASD participants. Together, we demonstrated the effectiveness of analyzing social gaze in people with ASD in naturalistic settings, showcasing the potential of third-person video perspectives in enhancing ASD diagnosis through gaze analysis.

Figures (7)

• Figure 1: Example video frames. (a, b) Caltech ADOS-2 Video Dataset. (c, d) WVU ADOS-2 Video Dataset.
• Figure 2: Example frames of visualizing the raw multimodal features. (a) Facial/gaze features extracted using the OpenFace algorithm. (b) Action features extracted using the OpenPose algorithm.
• Figure 3: Analysis pipeline. For each input video clip, we first separated the visual and audio tracks. The visual track, composed of all frames, and the audio track were further processed separately. Specifically, the visual track underwent feature detection using the OpenFace algorithm for facial/gaze features (Figure \ref{['raw:1']}) and the OpenPose algorithm for action features (Figure \ref{['raw:2']}), yielding raw multimodal features. The audio track was processed using Google's speech-to-text service for speech feature extraction. Subsequently, raw multimodal features were combined and processed to derive gaze metrics relevant to ASD based on domain knowledge of the condition. Ultimately, a random forest classifier was developed to differentiate between individuals exhibiting atypical and typical gaze patterns in the context of ASD.
• Figure 4: Summary of gaze metrics. (a) Gaze engagement ratio. (b) Gaze variance. (c) Gaze concentration area. (d) Gaze diversion frequency. Error bars denote ±SEM across samples. Each dot represents a sample. Asterisks indicate a significant difference using Welch's t-test. *: $p < 0.05$, and **: $p < 0.01$.
• Figure 5: Summary of gaze metrics during speaking and non-speaking intervals. (a-c) Speaking. (d-f) Non-speaking. (a, d) Gaze engagement ratio. (b, e) Gaze variance. (c, f) Gaze concentration area. Error bars denote ±SEM across samples. Each dot represents a sample. Asterisks indicate a significant difference using Welch's t-test. *: $p < 0.05$, and **: $p < 0.01$.