Designing LLM-simulated Immersive Spaces to Enhance Autistic Children's Social Affordances Understanding

TL;DR

Autistic children often struggle to infer social affordances in complex environments like traffic, impacting safety. The authors design AIRoad, an LLM-simulated immersive projection space guided by 17 design guidelines across General, Task, Interaction, and Information categories, to train understanding of driver intentions through multimodal cues. A within-subject study with 14 autistic children shows AIRoad improves usability, engagement, emotional state, and response timing, suggesting enhanced understanding of social affordances in traffic. The work demonstrates the potential of combining large language models with immersive environments for ASD rehabilitation, while outlining limitations (hallucinations, latency) and avenues for future multi-context extension and long-term transfer assessment.

Abstract

One of the key challenges faced by autistic children is understanding social affordances in complex environments, which further impacts their ability to respond appropriately to social signals. In traffic scenarios, this impairment can even lead to safety concerns. In this paper, we introduce an LLM-simulated immersive projection environment designed to improve this ability in autistic children while ensuring their safety. We first propose 17 design considerations across four major categories, derived from a comprehensive review of previous research. Next, we developed a system called AIroad, which leverages LLMs to simulate drivers with varying social intents, expressed through explicit multimodal social signals. AIroad helps autistic children bridge the gap in recognizing the intentions behind behaviors and learning appropriate responses through various stimuli. A user study involving 14 participants demonstrated that this technology effectively engages autistic children and leads to significant improvements in their comprehension of social affordances in traffic scenarios. Additionally, parents reported high perceived usability of the system. These findings highlight the potential of combining LLM technology with immersive environments for the functional rehabilitation of autistic children in the future.

Figures (11)

• Figure 1: The literature screening procedure in this study involved two rounds of evaluation. This process resulted in the retention of 74 articles as the basis for design considerations.
• Figure 2: System framework of AIRoad. The social affordance simulation conducted by the LLM is detailed in the social simulation module. The control of game rules by the LLM is outlined in the control module. Key prompts and outputs related to the LLM are also displayed below.
• Figure 3: AIRoad is an AI-enabled immersive educational space tailored for autistic children. It facilitates autistic children's learning about social affordances in complex traffic scenarios through the construction of virtual transportation settings.
• Figure 4: a) Projection on the ground; b) Projection on the wall; c) Overall real-life overview; d) Other cartoon elements; e) Creation of animation effects and vehicle driving style (30 frames per second).
• Figure 5: The experimental procedure of the study