Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Enabling American Sign Language Communication Under Low Data Rates

Panneer Selvam Santhalingam, Swann Thantsin, Ahmad Kamari, Parth Pathak, Kenneth DeHaan

TL;DR

VC4ASL tackles the challenge of ASL communication when video bandwidth is limited by encoding human pose information into the audio channel of existing video conferencing apps. It introduces a Chirp Spread Spectrum (CSS) modulation framework, frame-based design, and receive-side error detection/correction to transmit 2D pose keypoints efficiently (e.g., 56 bytes/frame with 112 symbols/frame) at up to ~6 kbps, enabling poses-per-second rates sufficient for ASL communication. The system uses a Mediapipe-derived pose pipeline, selective keypoint transmission, and a learned predictor for excluded joints, with autoencoder-based OOD detection to filter corrupted data and an error-correcting render stage. Extensive evaluations across Zoom and Meet show robust modulation under low-bandwidth conditions, effective pose reconstruction, and meaningful user comprehension, suggesting practical viability for ASL accessibility in bandwidth-constrained video conferencing. The work demonstrates a practical, compatible solution that enhances ASL accessibility without modifying existing VCAs, potentially informing broader accessibility tools and audio-based transmission strategies for visual languages.

Abstract

In recent years, video conferencing applications have become increasingly prevalent, relying heavily on high-speed internet connectivity. When such connectivity is lacking, users often default to audio-only communication, a mode that significantly disadvantages American Sign Language (ASL) users, whose communication relies on hand gestures, body movement, and facial expressions. In this work, we introduce VC4ASL, a system designed to enable ASL communication over the audio channel of existing video conferencing applications, even in the absence of reliable video. VC4ASL integrates seamlessly with current platforms without requiring any modifications. Our approach establishes a communication channel through audio by encoding and transmitting human pose information, which is then rendered to reconstruct signed content. We propose novel receive-side error detection and correction mechanisms that exploit the inherent structural constraints of human pose data. To evaluate the system, we simulate network-degraded environments, generate pose-based ASL video sequences, and conduct user studies to assess comprehension among ASL users. Experimental results demonstrate that VC4ASL effectively facilitates intelligible ASL communication over audio in low-bandwidth scenarios where video transmission is impaired.

Enabling American Sign Language Communication Under Low Data Rates

TL;DR

VC4ASL tackles the challenge of ASL communication when video bandwidth is limited by encoding human pose information into the audio channel of existing video conferencing apps. It introduces a Chirp Spread Spectrum (CSS) modulation framework, frame-based design, and receive-side error detection/correction to transmit 2D pose keypoints efficiently (e.g., 56 bytes/frame with 112 symbols/frame) at up to ~6 kbps, enabling poses-per-second rates sufficient for ASL communication. The system uses a Mediapipe-derived pose pipeline, selective keypoint transmission, and a learned predictor for excluded joints, with autoencoder-based OOD detection to filter corrupted data and an error-correcting render stage. Extensive evaluations across Zoom and Meet show robust modulation under low-bandwidth conditions, effective pose reconstruction, and meaningful user comprehension, suggesting practical viability for ASL accessibility in bandwidth-constrained video conferencing. The work demonstrates a practical, compatible solution that enhances ASL accessibility without modifying existing VCAs, potentially informing broader accessibility tools and audio-based transmission strategies for visual languages.

Abstract

In recent years, video conferencing applications have become increasingly prevalent, relying heavily on high-speed internet connectivity. When such connectivity is lacking, users often default to audio-only communication, a mode that significantly disadvantages American Sign Language (ASL) users, whose communication relies on hand gestures, body movement, and facial expressions. In this work, we introduce VC4ASL, a system designed to enable ASL communication over the audio channel of existing video conferencing applications, even in the absence of reliable video. VC4ASL integrates seamlessly with current platforms without requiring any modifications. Our approach establishes a communication channel through audio by encoding and transmitting human pose information, which is then rendered to reconstruct signed content. We propose novel receive-side error detection and correction mechanisms that exploit the inherent structural constraints of human pose data. To evaluate the system, we simulate network-degraded environments, generate pose-based ASL video sequences, and conduct user studies to assess comprehension among ASL users. Experimental results demonstrate that VC4ASL effectively facilitates intelligible ASL communication over audio in low-bandwidth scenarios where video transmission is impaired.
Paper Structure (32 sections, 7 equations, 12 figures, 6 tables)

This paper contains 32 sections, 7 equations, 12 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Background and Motivation
  3. System Overview
  4. Sender side:
  5. Receiver side:
  6. Pose Processor:
  7. Pose-to-Audio Encoder:
  8. Audio Codecs in VCA
  9. Designing a Robust Modulation Scheme
  10. Codec's impact on different modulation techniques
  11. $\mathsf{VC4ASL}$’s Modulation Scheme and Frame Design
  12. Audio-to-Pose Decoder
  13. Demodulation and symbol unpacking
  14. Pose Renderer
  15. Excluded Keypoint prediction
  16. ...and 17 more sections

Figures (12)

  • Figure 1: Impact of network conditions on different video conferencing applications
  • Figure 2: $\mathsf{VC4ASL}$'s different components.
  • Figure 3: Upper body and hand keypoints provided by Mediapipe and the joints transmitted by $\mathsf{VC4ASL}$
  • Figure 4: Comparing Frequency Shift Keying (FSK) and Chirp Spread Spectrum (CSS). (a) Different FSK and CSS symbols frequency response with and without codec compression. (b) The impact of data rate on the average SER averaged for different codec configurations. (c) The impact of codec configurations on the average SER averaged for different data rates.
  • Figure 5: $\mathsf{VC4ASL}$'s frame design showing the different parts of the frame and their durations.
  • ...and 7 more figures