Lost in Transcription: How Speech-to-Text Errors Derail Code Understanding
Jayant Havare, Ashish Mittal, Srikanth Tamilselvam, Ganesh Ramakrishnan
TL;DR
This work tackles the challenge of multilingual voice-driven code understanding by revealing how ASR transcription errors derail downstream reasoning over code. It introduces a modular cascade that combines multilingual ASR, LLM-guided code-aware transcription refinement, and code-model reasoning to support code question answering and code retrieval, delivering results in text and speech. Empirical evaluation across CodeQA, CodeSearchNet, and CoRNStack in Hindi, Gujarati, Tamil, Bengali, and English shows that transcription errors significantly hinder performance, but LLM-guided refinement yields substantial gains in both transcription fidelity ($WER$, $PER$, $WFED$) and downstream tasks (Recall@k, MRR). The findings underscore the necessity of code-sensitive adaptations in speech interfaces and demonstrate a practical pathway toward robust, multilingual voice-driven programming tools for education. The work also discusses challenges and threats to validity, and outlines future directions including domain-adapted ASR fine-tuning, expanded language support, and integration with classroom tooling to broaden accessibility.
Abstract
Code understanding is a foundational capability in software engineering tools and developer workflows. However, most existing systems are designed for English-speaking users interacting via keyboards, which limits accessibility in multilingual and voice-first settings, particularly in regions like India. Voice-based interfaces offer a more inclusive modality, but spoken queries involving code present unique challenges due to the presence of non-standard English usage, domain-specific vocabulary, and custom identifiers such as variable and function names, often combined with code-mixed expressions. In this work, we develop a multilingual speech-driven framework for code understanding that accepts spoken queries in a user native language, transcribes them using Automatic Speech Recognition (ASR), applies code-aware ASR output refinement using Large Language Models (LLMs), and interfaces with code models to perform tasks such as code question answering and code retrieval through benchmarks such as CodeSearchNet, CoRNStack, and CodeQA. Focusing on four widely spoken Indic languages and English, we systematically characterize how transcription errors impact downstream task performance. We also identified key failure modes in ASR for code and demonstrated that LLM-guided refinement significantly improves performance across both transcription and code understanding stages. Our findings underscore the need for code-sensitive adaptations in speech interfaces and offer a practical solution for building robust, multilingual voice-driven programming tools.