Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Understanding the Use of a Large Language Model-Powered Guide to Make Virtual Reality Accessible for Blind and Low Vision People

Jazmin Collins, Sharon Y Lin, Tianqi Liu, Andrea Stevenson Won, Shiri Azenkot

TL;DR

A large language model (LLM)-powered guide is developed and studied its use with 16 BLV participants in virtual environments with confederates posing as other users, finding that when alone, participants treated the guide as a tool, but treated it companionably around others.

Abstract

As social virtual reality (VR) grows more popular, addressing accessibility for blind and low vision (BLV) users is increasingly critical. Researchers have proposed an AI "sighted guide" to help users navigate VR and answer their questions, but it has not been studied with users. To address this gap, we developed a large language model (LLM)-powered guide and studied its use with 16 BLV participants in virtual environments with confederates posing as other users. We found that when alone, participants treated the guide as a tool, but treated it companionably around others, giving it nicknames, rationalizing its mistakes with its appearance, and encouraging confederate-guide interaction. Our work furthers understanding of guides as a versatile method for VR accessibility and presents design recommendations for future guides.

Understanding the Use of a Large Language Model-Powered Guide to Make Virtual Reality Accessible for Blind and Low Vision People

TL;DR

A large language model (LLM)-powered guide is developed and studied its use with 16 BLV participants in virtual environments with confederates posing as other users, finding that when alone, participants treated the guide as a tool, but treated it companionably around others.

Abstract

As social virtual reality (VR) grows more popular, addressing accessibility for blind and low vision (BLV) users is increasingly critical. Researchers have proposed an AI "sighted guide" to help users navigate VR and answer their questions, but it has not been studied with users. To address this gap, we developed a large language model (LLM)-powered guide and studied its use with 16 BLV participants in virtual environments with confederates posing as other users. We found that when alone, participants treated the guide as a tool, but treated it companionably around others, giving it nicknames, rationalizing its mistakes with its appearance, and encouraging confederate-guide interaction. Our work furthers understanding of guides as a versatile method for VR accessibility and presents design recommendations for future guides.
Paper Structure (25 sections, 4 figures, 3 tables)

This paper contains 25 sections, 4 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Enhancing the Accessibility of Social VR Environments
  4. Supporting BLV People with Guides in the Physical World
  5. Supporting BLV People with Visual Interpretation in the Physical World
  6. Methods
  7. Participants
  8. Procedure
  9. Prototype Design and Implementation
  10. Data and Analysis
  11. Findings
  12. Overview
  13. Participants’ Ratings of the Guide and their VR Experience
  14. Completing Tasks with the Guide
  15. Exploring Alone
  16. ...and 10 more sections

Figures (4)

  • Figure 1: The three virtual environments used in the study. From left to right, the Tutorial contains various simplistic 3D objects such as yellow and green cubes, Park 1 contains a large river and several colorful gazebos, and Park 2 contains colorful flower beds and a series of benches for resting throughout the park.
  • Figure 2: Types of requests made to the guide. Participants made a total of 799 individual requests within 467 user queries, as many queries encompassed multiple request types. For instance, the query “Describe where that sound is coming from” includes both a visual description and an auditory description.
  • Figure 3: Tones of requests made to the guide. Participants made 476 requests total, one for each query to the guide, as each query counted as a separate utterance with its own tone.
  • Figure 4: Likert scale responses from participants rating the guide, collected after participants experienced both parks and guide personas. Scores were given on a scale of 1 to 5, where 1 = strongly disagree, 2 = disagree, 3 = neither agree nor disagree, 4 = agree, and 5 = strongly agree. Note: In the statement on Social Comfort, “others” refers to the confederates participants met during the study. We only include scores from thirteen participants who used the guide.