Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Development of Immersive Virtual and Augmented Reality-Based Joint Attention Training Platform for Children with Autism

Ashirbad Samantaray, Taranjit Kaur, Sapna S Mishra, Kritika Lohia, Chayan Majumder, Sheffali Gulati, Tapan Kumar Gandhi

TL;DR

The study develops a dual immersive JA training platform for children with ASD using VR and AR with integrated eye tracking and an avatar mediator. It conducts a feasibility comparison between ASD and NT groups across VR and AR, finding NT participants respond faster in both settings while ASD participants achieve high accuracy, with AR showing a stronger link between autism severity and response latency. The results support the platform's feasibility and potential, particularly AR, for early JA intervention, while signaling the need for larger, longitudinal studies and real-world validation. Overall, the work establishes a foundation for gaze-driven, avatar-mediated JA training across immersive modalities.

Abstract

Joint Attention (JA), a crucial social skill for developing shared focus, is often impaired in children with Autism Spectrum Disorder (ASD), affecting social communication and highlighting the need for early intervention. Addressing gaps in prior research, such as limited use of immersive technology and reliance on distracting peripherals, we developed a novel JA training platform using Augmented Reality (AR) and Virtual Reality (VR) devices. The platform integrates eye gaze-based interactions to ensure participants undivided attention. To validate the platform, we conducted experiments on ASD (N=19) and Neurotypical (NT) (N=13) participants under a trained pediatric neurologist's supervision. For quantitative analysis, we employed key measures such as the number of correct responses, the duration of establishing eye contact (s), and the duration of registering a response (s), along with correlations to CARS scores and age. Results from AR-based experiments showed NT participants registered responses significantly faster (<0.00001) than ASD participants. A correlation (Spearman coefficient=0.57, p=0.03) was found between ASD participants response time and CARS scores. A similar trend was observed in VR-based experiments. When comparing response accuracy in ASD participants across platforms, AR yielded a higher correctness rate (92.30%) than VR (69.49%), indicating AR's greater effectiveness. These findings suggest that immersive technology can aid JA training in ASD. Future studies should explore long-term benefits and real-world applicability.

Development of Immersive Virtual and Augmented Reality-Based Joint Attention Training Platform for Children with Autism

TL;DR

The study develops a dual immersive JA training platform for children with ASD using VR and AR with integrated eye tracking and an avatar mediator. It conducts a feasibility comparison between ASD and NT groups across VR and AR, finding NT participants respond faster in both settings while ASD participants achieve high accuracy, with AR showing a stronger link between autism severity and response latency. The results support the platform's feasibility and potential, particularly AR, for early JA intervention, while signaling the need for larger, longitudinal studies and real-world validation. Overall, the work establishes a foundation for gaze-driven, avatar-mediated JA training across immersive modalities.

Abstract

Joint Attention (JA), a crucial social skill for developing shared focus, is often impaired in children with Autism Spectrum Disorder (ASD), affecting social communication and highlighting the need for early intervention. Addressing gaps in prior research, such as limited use of immersive technology and reliance on distracting peripherals, we developed a novel JA training platform using Augmented Reality (AR) and Virtual Reality (VR) devices. The platform integrates eye gaze-based interactions to ensure participants undivided attention. To validate the platform, we conducted experiments on ASD (N=19) and Neurotypical (NT) (N=13) participants under a trained pediatric neurologist's supervision. For quantitative analysis, we employed key measures such as the number of correct responses, the duration of establishing eye contact (s), and the duration of registering a response (s), along with correlations to CARS scores and age. Results from AR-based experiments showed NT participants registered responses significantly faster (<0.00001) than ASD participants. A correlation (Spearman coefficient=0.57, p=0.03) was found between ASD participants response time and CARS scores. A similar trend was observed in VR-based experiments. When comparing response accuracy in ASD participants across platforms, AR yielded a higher correctness rate (92.30%) than VR (69.49%), indicating AR's greater effectiveness. These findings suggest that immersive technology can aid JA training in ASD. Future studies should explore long-term benefits and real-world applicability.

Paper Structure

This paper contains 17 sections, 19 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Materials and Methodology
  4. VR Setup (FOVE 0 Headset)
  5. AR Setup (HoloLens 2 Headset)
  6. Software
  7. Experiment Design
  8. Experiment Protocol
  9. Task Description
  10. Participants
  11. Results
  12. Statistical Group Differences of Task Performance Metrics
  13. Correlation Analysis between Task Performance Metrics and Participant Characteristics
  14. Comparison of Correctness of Response (CPR) across the participant groups
  15. Discussion
  16. ...and 2 more sections

Figures (19)

  • Figure 1: Participant wearing FOVE 0 VR headset for JA training.
  • Figure 2: Participant wearing HoloLens 2 AR headset for JA training
  • Figure 3: System Design Overview of JA Training Platform. The task environment designed in Unity delivers JA exercises with modules for gaze fixation registration and task mediation. An experiment controller developed using Node.js coordinates data flow between the experimenter interface. It logs task performance and mirrors the participant's view for monitoring. Hardware components include an experimenter's laptop, a Microsoft HoloLens 2 AR device, and a FOVE 0 VR device.
  • Figure 4: Process pipeline for the design and development of an avatar as a task mediator in our JA application
  • Figure 5: Cue Delivery Sequence of Avatar in VR platform (A) Overview of VR environment in Unity Editor (B) Waiting to establish eye contact (C) Delivery of finger pointing cue by avatar to right and left (D) side
  • ...and 14 more figures