CodeVaani: A Multilingual, Voice-Based Code Learning Assistant

TL;DR

CodeVaani addresses the barrier of English-centric, text-based programming education by delivering a multilingual, speech-driven assistant integrated into BodhiTree. The system pipes native-language speech through ASR (Whisper for English, Indic-Conformer for Indian languages), refines transcripts with a code-aware module (gemma-27B + DPO), and generates code-focused responses via Codestral-22B, providing text and audio outputs. In a study with 28 beginners, CodeVaani achieved $75\%$ accuracy and received positive reactions from over $80\%$ of participants, demonstrating potential for on-demand, scalable, multilingual programming support. These results suggest that voice-based, language-inclusive tools can meaningfully expand access to programming education in multilingual regions and in classroom settings.

Abstract

Programming education often assumes English proficiency and text-based interaction, creating barriers for students from multilingual regions such as India. We present CodeVaani, a multilingual speech-driven assistant for understanding code, built into Bodhitree [1], a Learning Management System developed at IIT Bombay. It is a voice-enabled assistant that helps learners explore programming concepts in their native languages. The system integrates Indic ASR, a codeaware transcription refinement module, and a code model for generating relevant answers. Responses are provided in both text and audio for natural interaction. In a study with 28 beginner programmers, CodeVaani achieved 75% response accuracy, with over 80% of participants rating the experience positively. Compared to classroom assistance, our framework offers ondemand availability, scalability to support many learners, and multilingual support that lowers the entry barrier for students with limited English proficiency. The demo will illustrate these capabilities and highlight how voice-based AI systems can make programming education more inclusive. Supplementary artifacts and demo video are also made available.

Figures (4)

• Figure 1: Survey results from 53 engineering students across multiple Indian states, highlighting the multilingual context of programming education.
• Figure 2: Architecture of CodeVaani: Processing student queries through ASR, error correction, and AI code generation.
• Figure 3: Example of Hindi code-related spoken query and ASR transcriptions. Errors highlight the challenges of code-mixed speech.
• Figure 4: Demo interface where participant’s query was transcribed by ASR, refined by error correction (shown in What we heard field), and answered in the CodeVaani Response field