Sanvaad: A Multimodal Accessibility Framework for ISL Recognition and Voice-Based Interaction

TL;DR

Sanvaad presents a lightweight, edge-deployable multimodal framework that enables real-time two-way communication for deaf and visually impaired users by integrating an ISL recognition module (MediaPipe-based landmarks with a residual MLP), a voice-to-sign translation, and a voice translator for multilingual news summaries. The system achieves 84% ISL classification accuracy with 35 classes, under 500 ms V2S latency, and ROUGE-L scores above 0.45 for summaries, demonstrating practical viability on commodity hardware. Its design emphasizes offline, sensor-free operation, Streamlit-based accessibility, and careful data augmentation to cope with ISL data scarcity, noise, and variation. The work also outlines concrete avenues for richer linguistic context, multilingual expansion, and mobile/offline deployment to broaden impact in India and similar contexts.

Abstract

Communication between deaf users, visually im paired users, and the general hearing population often relies on tools that support only one direction of interaction. To address this limitation, this work presents Sanvaad, a lightweight multimodal accessibility framework designed to support real time, two-way communication. For deaf users, Sanvaad includes an ISL recognition module built on MediaPipe landmarks. MediaPipe is chosen primarily for its efficiency and low computational load, enabling the system to run smoothly on edge devices without requiring dedicated hardware. Spoken input from a phone can also be translated into sign representations through a voice-to-sign component that maps detected speech to predefined phrases and produces corresponding GIFs or alphabet-based visualizations. For visually impaired users, the framework provides a screen free voice interface that integrates multilingual speech recognition, text summarization, and text-to-speech generation. These components work together through a Streamlit-based interface, making the system usable on both desktop and mobile environments. Overall, Sanvaad aims to offer a practical and accessible pathway for inclusive communication by combining lightweight computer vision and speech processing tools within a unified framework.

Figures (7)

• Figure 1: Application architecture
• Figure 2: Sample distribution per class, 1-9 for numbers and A-Z for letters
• Figure 3: Dataset images and MediaPipe landmark extraction visualised
• Figure 4: 141 landmark distribution, 126 from hands, 15 augmented
• Figure 5: Model Architecture