Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Design Considerations for Automatic Musical Soundscapes of Visual Art for People with Blindness or Low Vision

Stephen James Krol, Maria Teresa Llano, Matthew Butler, Cagatay Goncu

TL;DR

The paper tackles the challenge of making visual art accessible to blind and low-vision audiences by automating musical soundscapes that convey mood and context. It introduces a two-stage prototype system that first generates emotive music from paintings and then augments it with contextual sound effects, guided by object detection; the system is evaluated through a pilot with sighted and BLV participants and an in-depth qualitative study with BLV participants. Key contributions include an emotion-classification pipeline mapped to the Valence-Arousal space, a Magenta Music Transformer-based music generator, and a context-aware sound-effects database, all analyzed to derive design considerations emphasizing narrative feel, diversity and accuracy of contextual cues, and integration with other accessibility methods. The findings suggest automated soundscapes can produce meaningful aesthetic experiences for BLV users while highlighting design principles such as narrative framing, customization, historical accuracy, ambient immersion, and careful audio-reproduction to ensure representativeness and minimize misrepresentation, with potential for scalable museum deployment.

Abstract

Music has been identified as a promising medium to enhance the accessibility and experience of visual art for people who are blind or have low vision (BLV). However, composing music and designing soundscapes for visual art is a time-consuming, resource intensive process - limiting its scalability for large exhibitions. In this paper, we investigate the use of automated soundscapes to increase the accessibility of visual art. We built a prototype system and ran a qualitative study to evaluate the aesthetic experience provided by the automated soundscapes with 10 BLV participants. From the study, we identified a set of design considerations that reveal requirements from BLV people for the development of automated soundscape systems, setting new directions in which creative systems could enrich the aesthetic experience conveyed by these.

Design Considerations for Automatic Musical Soundscapes of Visual Art for People with Blindness or Low Vision

TL;DR

The paper tackles the challenge of making visual art accessible to blind and low-vision audiences by automating musical soundscapes that convey mood and context. It introduces a two-stage prototype system that first generates emotive music from paintings and then augments it with contextual sound effects, guided by object detection; the system is evaluated through a pilot with sighted and BLV participants and an in-depth qualitative study with BLV participants. Key contributions include an emotion-classification pipeline mapped to the Valence-Arousal space, a Magenta Music Transformer-based music generator, and a context-aware sound-effects database, all analyzed to derive design considerations emphasizing narrative feel, diversity and accuracy of contextual cues, and integration with other accessibility methods. The findings suggest automated soundscapes can produce meaningful aesthetic experiences for BLV users while highlighting design principles such as narrative framing, customization, historical accuracy, ambient immersion, and careful audio-reproduction to ensure representativeness and minimize misrepresentation, with potential for scalable museum deployment.

Abstract

Music has been identified as a promising medium to enhance the accessibility and experience of visual art for people who are blind or have low vision (BLV). However, composing music and designing soundscapes for visual art is a time-consuming, resource intensive process - limiting its scalability for large exhibitions. In this paper, we investigate the use of automated soundscapes to increase the accessibility of visual art. We built a prototype system and ran a qualitative study to evaluate the aesthetic experience provided by the automated soundscapes with 10 BLV participants. From the study, we identified a set of design considerations that reveal requirements from BLV people for the development of automated soundscape systems, setting new directions in which creative systems could enrich the aesthetic experience conveyed by these.
Paper Structure (38 sections, 4 figures, 1 table)

This paper contains 38 sections, 4 figures, 1 table.

Table of Contents

  1. Introduction
  2. Background
  3. Accessibility of Visual Art
  4. Current Accessibility Methods
  5. Accessibility of the Aesthetic Experience
  6. Generating Sound from Images
  7. Prototype System Description
  8. Stage 1: Generating Music with Emotion
  9. Emotion Classification
  10. Emotionally Annotated Music Database
  11. Music Generation
  12. Stage 2: Incorporating Contextual Information
  13. Object Detection
  14. Sound Effects Database
  15. Pilot Study
  16. ...and 23 more sections

Figures (4)

  • Figure 1: System Diagram: Depicts the various tasks performed by the system to generate the output music.
  • Figure 2: Emotional Mappings: The above figure shows the mappings used to convert the categorical emotions in WikiArt to emotions in the Valence-Arousal scale.
  • Figure 3: Paintings used in the study. 1: Tennis Triptych Centre Panel, 2: A Cow in a Landscape, 3: A Saddled Horse Tied to a Fence, 4: James Abercrombe of Tullibody esq, 5: Hip Hip Hurrah, 6: Herdsmen with Resting Cattle, and 7: Paris a Rainy Day.
  • Figure 4: Themes identified through the thematic analysis of participants' interviews.