ArtInsight: Enabling AI-Powered Artwork Engagement for Mixed Visual-Ability Families

TL;DR

This paper tackles the challenge of enabling BLV family members to engage with their children's artwork by introducing ArtInsight, an AI-powered system that provides descriptive and creative interpretations, records the child’s narration, and generates discussion questions. The authors present a custom prompt-engineered GPT-4o backend, a user-centric interface with an AI personality toggle, audio augmentation, and an in-app question module, plus a rubric for evaluating AI artwork descriptions. Through a mixed-methods evaluation with BLV families and a case study with a blind therapist, ArtInsight outperformed a popular BLV tool in perceived usefulness, with the audio and child-narrative components driving perceived value and engagement. The findings offer design guidelines for accessible, context-aware AI in mixed-ability settings and highlight ethical considerations around privacy and data handling in pediatric AI applications, suggesting a path toward broader applications in family interactions and art therapy.

Abstract

We introduce ArtInsight, a novel AI-powered system to facilitate deeper engagement with child-created artwork in mixed visual-ability families. ArtInsight leverages large language models (LLMs) to craft a respectful and thorough initial description of a child's artwork, and provides: creative AI-generated descriptions for a vivid overview, audio recording to capture the child's own description of their artwork, and a set of AI-generated questions to facilitate discussion between blind or low-vision (BLV) family members and their children. Alongside ArtInsight, we also contribute a new rubric to score AI-generated descriptions of child-created artwork and an assessment of state-of-the-art LLMs. We evaluated ArtInsight with five groups of BLV family members and their children, and as a case study with one BLV child therapist. Our findings highlight a preference for ArtInsight's longer, artistically-tailored descriptions over those generated by existing BLV AI tools. Participants highlighted the creative description and audio recording components as most beneficial, with the former helping ``bring a picture to life'' and the latter centering the child's narrative to generate context-aware AI responses. Our findings reveal different ways that AI can be used to support art engagement, including before, during, and after interaction with the child artist, as well as expectations that BLV adults and their sighted children have about AI-powered tools.

Figures (8)

• Figure 1: ArtInsight allows BLV family members to (A) take photos of child-created artwork, (B) hear AI-generated descriptions of the artwork, (C) record and use an audio recording of the child describing their artwork for added context sent to the model, and (D) parse three AI-generated starter questions for dialogue and discussion with children.
• Figure 2: The end-to-end workflow of navigating to the audio recording component, capturing an audio snippet, and using the transcript of the snippet to update the AI description of the artwork.
• Figure 3: Participants' Perceived Accuracy (not statistically significant) and Usefulness (statistically significant) ratings for initial AI descriptions of Be My AI.
• Figure 4: Example artwork by P4's son (left) and P1's daughter (right) used during the comparison portion of the studies.
• Figure 5: P2's daughter's incomplete drawing of a person, along with snippets of the initial ArtInsight and Be My AI descriptions. Neither description mentioned the gradient of the hair, which P2's daughter wanted to surface for her father.