Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

LAHAJA: A Robust Multi-accent Benchmark for Evaluating Hindi ASR Systems

Tahir Javed, Janki Nawale, Sakshi Joshi, Eldho George, Kaushal Bhogale, Deovrat Mehendale, Mitesh M. Khapra

TL;DR

LAHAJA addresses the challenge of Hindi ASR across diverse accents by introducing a 12.5-hour, multi-dialect benchmark compiled from 132 speakers across 83 districts, spanning 19 native languages and 4 language families. The authors evaluate state-of-the-art baselines and propose IndicASR variants trained on varied multilingual Hindi data, including synthetic code-mixed content, achieving a notable reduction in Word Error Rate (WER) to 14.3% with the M3 model. A fine-grained analysis reveals accent-specific gaps, with North-East and South Indian speakers and content rich in named entities being more challenging. The work provides public datasets, models, and scripts to foster further research on robust, multi-accent Hindi ASR systems with broad speaker diversity and domain coverage.

Abstract

Hindi, one of the most spoken language of India, exhibits a diverse array of accents due to its usage among individuals from diverse linguistic origins. To enable a robust evaluation of Hindi ASR systems on multiple accents, we create a benchmark, LAHAJA, which contains read and extempore speech on a diverse set of topics and use cases, with a total of 12.5 hours of Hindi audio, sourced from 132 speakers spanning 83 districts of India. We evaluate existing open-source and commercial models on LAHAJA and find their performance to be poor. We then train models using different datasets and find that our model trained on multilingual data with good speaker diversity outperforms existing models by a significant margin. We also present a fine-grained analysis which shows that the performance declines for speakers from North-East and South India, especially with content heavy in named entities and specialized terminology.

LAHAJA: A Robust Multi-accent Benchmark for Evaluating Hindi ASR Systems

TL;DR

LAHAJA addresses the challenge of Hindi ASR across diverse accents by introducing a 12.5-hour, multi-dialect benchmark compiled from 132 speakers across 83 districts, spanning 19 native languages and 4 language families. The authors evaluate state-of-the-art baselines and propose IndicASR variants trained on varied multilingual Hindi data, including synthetic code-mixed content, achieving a notable reduction in Word Error Rate (WER) to 14.3% with the M3 model. A fine-grained analysis reveals accent-specific gaps, with North-East and South Indian speakers and content rich in named entities being more challenging. The work provides public datasets, models, and scripts to foster further research on robust, multi-accent Hindi ASR systems with broad speaker diversity and domain coverage.

Abstract

Hindi, one of the most spoken language of India, exhibits a diverse array of accents due to its usage among individuals from diverse linguistic origins. To enable a robust evaluation of Hindi ASR systems on multiple accents, we create a benchmark, LAHAJA, which contains read and extempore speech on a diverse set of topics and use cases, with a total of 12.5 hours of Hindi audio, sourced from 132 speakers spanning 83 districts of India. We evaluate existing open-source and commercial models on LAHAJA and find their performance to be poor. We then train models using different datasets and find that our model trained on multilingual data with good speaker diversity outperforms existing models by a significant margin. We also present a fine-grained analysis which shows that the performance declines for speakers from North-East and South India, especially with content heavy in named entities and specialized terminology.
Paper Structure (10 sections, 3 figures, 6 tables)

This paper contains 10 sections, 3 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Lahaja
  3. Recruitment of Speakers
  4. Data collection
  5. Transcription
  6. Statistics
  7. Experimental Setup
  8. Results and Discussion
  9. Conclusion
  10. Acknowledgements

Figures (3)

  • Figure 1: Different districts of India from which data was collected. The colors show how the WER of our best model varies across different regions of India (shades of green being relatively better and shades of red and brown being relatively poor).
  • Figure 2: Demographic distribution of participants in Lahaja across age group, job segment and educational background.
  • Figure 3: Performance breakdown of M3 & Google Chirp across non-native speakers.