Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

VRSL:Exploring the Comprehensibility of 360-Degree Camera Feeds for Sign Language Communication in Virtual Reality

Gauri Umesh Rajmane, Ziming Li, Tae Oh, Roshan Peiris

TL;DR

Examination of the comprehensibility and user experience of viewing American Sign Language videos captured with body-mounted 360-degree cameras showed the shoulder-mounted camera achieved the highest accuracy (85%), though differences between positions were not statistically significant.

Abstract

This study explores integrating sign language into virtual reality (VR) by examining the comprehensibility and user experience of viewing American Sign Language (ASL) videos captured with body-mounted 360-degree cameras. Ten participants identified ASL signs from videos recorded at three body-mounted positions: head, shoulder, and chest. Results showed the shoulder-mounted camera achieved the highest accuracy (85%), though differences between positions were not statistically significant. Participants noted that peripheral distortion in 360-degree videos impacted clarity, highlighting areas for improvement. Despite challenges, the overall comprehension success rate of 83.3% demonstrates the potential of video-based ASL communication in VR. Feedback emphasized the need to refine camera angles, reduce distortion, and explore alternative mounting positions. Participants expressed a preference for signing over text-based communication in VR, highlighting the importance of developing this approach to enhance accessibility and collaboration for Deaf and Hard of Hearing (DHH) users in virtual environments.

VRSL:Exploring the Comprehensibility of 360-Degree Camera Feeds for Sign Language Communication in Virtual Reality

TL;DR

Examination of the comprehensibility and user experience of viewing American Sign Language videos captured with body-mounted 360-degree cameras showed the shoulder-mounted camera achieved the highest accuracy (85%), though differences between positions were not statistically significant.

Abstract

This study explores integrating sign language into virtual reality (VR) by examining the comprehensibility and user experience of viewing American Sign Language (ASL) videos captured with body-mounted 360-degree cameras. Ten participants identified ASL signs from videos recorded at three body-mounted positions: head, shoulder, and chest. Results showed the shoulder-mounted camera achieved the highest accuracy (85%), though differences between positions were not statistically significant. Participants noted that peripheral distortion in 360-degree videos impacted clarity, highlighting areas for improvement. Despite challenges, the overall comprehension success rate of 83.3% demonstrates the potential of video-based ASL communication in VR. Feedback emphasized the need to refine camera angles, reduce distortion, and explore alternative mounting positions. Participants expressed a preference for signing over text-based communication in VR, highlighting the importance of developing this approach to enhance accessibility and collaboration for Deaf and Hard of Hearing (DHH) users in virtual environments.
Paper Structure (22 sections, 3 figures)

This paper contains 22 sections, 3 figures.

Table of Contents

  1. Introduction
  2. Prior Work
  3. Communication in Virtual Reality
  4. DHH Experience in VR Communication
  5. Sign Language and Collaboration in VR
  6. Closed Captioning
  7. Sound Accessibility
  8. Hand-Tracking and Sign Language Recognition
  9. Text-Based Communication
  10. Methodology
  11. Study Design
  12. Procedure
  13. Participants
  14. Apparatus: Video Capture and Camera Mounting Positions
  15. Shoulder-Mounted Position
  16. ...and 7 more sections

Figures (3)

  • Figure 1: Camera mounting positions and their view. Encircled are the location of the camera. Top row shows the shoulder-mounted (a-top), chest-mounted (b-top) and head-mounted cameras (c-top) on the signer. Bottom row shows the footage captured by the shoulder (a-bottom), chest (b-bottom) and head (c-bottom) mounted cameras
  • Figure 2: Screenshots of the VR study software: (a) Welcome screen in the VR Application (b) participant on-boarding and instructions on how to use the controller settings and navigate. (c) Options for user to select where they want To view the video, (d) Demo for the user to see a bottom-right video viewing position, (e) Task for user to interpret video being signed for the word "calculator"
  • Figure 3: (a): Average accuracy percentage and (b): TLX scores across mental, physical, temporal, Success, Effort and Frustration for the three conditions