Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

MusicTraces: A collaborative music and paint activity for autistic people

Valentin Bauer, Tommaso Padovano, Mattia Gianotti, Giacomo Caslini, Franca Garzotto

TL;DR

This work investigates how Multisensory Environments can support collaborative art-making for autistic individuals by designing MusicTraces, a two-user full-body music-and-paint activity embedded in an MSE with smart objects and a Kinect-based setup. Through a multi-phase co-design process—including a workshop, initial design, remote iterations, and a design critique—the authors elicit caregiver and participant insights to shape the environment, syntax, interactions, hint system, and tablet interface. Key contributions include a unique two-user, open-ended integration of music and painting within an MSE, a detailed design and validation process, and a roadmap for portability and empirical evaluation. The study highlights the potential of co-designed, multisensory, collaborative activities to promote social interaction and creativity among individuals with moderate to severe neurodevelopmental conditions, while outlining practical steps for broader deployment and evaluation.

Abstract

Painting and music therapy approaches can help to foster social interaction for autistic people. However, the tools sometimes lack of flexibility and fail to keep people's attention. Unknowns also remain about the effect of combining these approaches. Though, very few studies have investigated how Multisensory Environments (MSEs) could help to address these issues. This paper presents the design of a full-body music and painting activity called "MusicTraces" which aims to foster collaboration between people with moderate to severe learning disabilities and complex needs, and in particular autism, within an MSE. The co-design process with caregivers and people neurodevelopmental conditions is detailed, including a workshop, the initial design, remote iterations, and a design critique.

MusicTraces: A collaborative music and paint activity for autistic people

TL;DR

This work investigates how Multisensory Environments can support collaborative art-making for autistic individuals by designing MusicTraces, a two-user full-body music-and-paint activity embedded in an MSE with smart objects and a Kinect-based setup. Through a multi-phase co-design process—including a workshop, initial design, remote iterations, and a design critique—the authors elicit caregiver and participant insights to shape the environment, syntax, interactions, hint system, and tablet interface. Key contributions include a unique two-user, open-ended integration of music and painting within an MSE, a detailed design and validation process, and a roadmap for portability and empirical evaluation. The study highlights the potential of co-designed, multisensory, collaborative activities to promote social interaction and creativity among individuals with moderate to severe neurodevelopmental conditions, while outlining practical steps for broader deployment and evaluation.

Abstract

Painting and music therapy approaches can help to foster social interaction for autistic people. However, the tools sometimes lack of flexibility and fail to keep people's attention. Unknowns also remain about the effect of combining these approaches. Though, very few studies have investigated how Multisensory Environments (MSEs) could help to address these issues. This paper presents the design of a full-body music and painting activity called "MusicTraces" which aims to foster collaboration between people with moderate to severe learning disabilities and complex needs, and in particular autism, within an MSE. The co-design process with caregivers and people neurodevelopmental conditions is detailed, including a workshop, the initial design, remote iterations, and a design critique.
Paper Structure (20 sections, 2 figures)

This paper contains 20 sections, 2 figures.

Table of Contents

  1. Introduction
  2. Related Work
  3. Phase 1 - Co-design workshop
  4. Participants
  5. Protocol
  6. Findings
  7. Phase 2: Initial design
  8. Environment of the activity
  9. The MusicTraces Syntax
  10. Interactions
  11. Hint system
  12. Tablet interface
  13. Apparatus
  14. Phase 3 - Remote agile process
  15. Method
  16. ...and 5 more sections

Figures (2)

  • Figure 1: Examples of the material used during the co-design workshop. (a.) Example of activity-related card, (b.) Example of design heuristic card, (c.) Example of answers on a "concept sheet" from a participant with neurodevelopmental condition (PwD1).
  • Figure 2: Design of the tablet user interface (UI), smart brush, and smart eraser. Elements added after the design phase 4 are mentioned with (D4). (a.) Tablet UI. The Left panel contains buttons to stop the game, control the tutorial steps, remove lines/circles, activate the background music, activate the music evolution (D4), play all melodies, activate the blobs, swap players or hands (security). The center panel manages the hints. The right panel displays the notifications. It can be hidden by clicking on the column at its left (D4). (b.) Smart brush used to draw when pushing on the rainbow button. c. Smart eraser used to erase when holding it.