Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Cost Analysis of Human-corrected Transcription for Predominately Oral Languages

Yacouba Diarra, Nouhoum Souleymane Coulibaly, Michael Leventhal

TL;DR

This paper tackles the cost of creating NLP resources for Predominately Oral Languages by conducting a one-month field study in Bambara with ten native-level transcribers who post-edit ASR-generated transcripts for 53 hours of speech. The authors implement a first-review transcription workflow using Label Studio, VAD-based segmentation, and pre-labeled ASR outputs to measure human labor, achieving a labor cost of about 30 hours per hour of annotated audio under ideal conditions and roughly 36 hours per hour in field conditions. The study provides a concrete, empirically grounded baseline for budgeting and planning data collection for low-literacy POLs, highlighting the substantial cognitive and logistic demands that exceed those typical for higher-literacy languages. The findings have practical implications for resource allocation, pipeline design, and the evaluation of synthetic data approaches for building robust speech technologies in POLs. By clarifying the labor requirements, the work informs policy and practice for scalable NLP resource development in low-resource, predominantly oral language communities.

Abstract

Creating speech datasets for low-resource languages is a critical yet poorly understood challenge, particularly regarding the actual cost in human labor. This paper investigates the time and complexity required to produce high-quality annotated speech data for a subset of low-resource languages, low literacy Predominately Oral Languages, focusing on Bambara, a Manding language of Mali. Through a one-month field study involving ten transcribers with native proficiency, we analyze the correction of ASR-generated transcriptions of 53 hours of Bambara voice data. We report that it takes, on average, 30 hours of human labor to accurately transcribe one hour of speech data under laboratory conditions and 36 hours under field conditions. The study provides a baseline and practical insights for a large class of languages with comparable profiles undertaking the creation of NLP resources.

Cost Analysis of Human-corrected Transcription for Predominately Oral Languages

TL;DR

This paper tackles the cost of creating NLP resources for Predominately Oral Languages by conducting a one-month field study in Bambara with ten native-level transcribers who post-edit ASR-generated transcripts for 53 hours of speech. The authors implement a first-review transcription workflow using Label Studio, VAD-based segmentation, and pre-labeled ASR outputs to measure human labor, achieving a labor cost of about 30 hours per hour of annotated audio under ideal conditions and roughly 36 hours per hour in field conditions. The study provides a concrete, empirically grounded baseline for budgeting and planning data collection for low-literacy POLs, highlighting the substantial cognitive and logistic demands that exceed those typical for higher-literacy languages. The findings have practical implications for resource allocation, pipeline design, and the evaluation of synthetic data approaches for building robust speech technologies in POLs. By clarifying the labor requirements, the work informs policy and practice for scalable NLP resource development in low-resource, predominantly oral language communities.

Abstract

Creating speech datasets for low-resource languages is a critical yet poorly understood challenge, particularly regarding the actual cost in human labor. This paper investigates the time and complexity required to produce high-quality annotated speech data for a subset of low-resource languages, low literacy Predominately Oral Languages, focusing on Bambara, a Manding language of Mali. Through a one-month field study involving ten transcribers with native proficiency, we analyze the correction of ASR-generated transcriptions of 53 hours of Bambara voice data. We report that it takes, on average, 30 hours of human labor to accurately transcribe one hour of speech data under laboratory conditions and 36 hours under field conditions. The study provides a baseline and practical insights for a large class of languages with comparable profiles undertaking the creation of NLP resources.
Paper Structure (10 sections, 1 figure, 2 tables)

This paper contains 10 sections, 1 figure, 2 tables.

Table of Contents

  1. Introduction
  2. How much time to transcribe one hour of speech?
  3. Analysis of the Experiment
  4. Setup and Task Definition
  5. Transcribers Profile
  6. Performance Analysis
  7. Conclusion
  8. Limitations
  9. Ethics Statement
  10. Acknowledgements

Figures (1)

  • Figure 1: Transcription Effort Spectrum: Human Labor Multiplier per Language Category. The values on the top represent the time factor for transcribing an hour of audio against language community literacy labeled below the chart. We do a simple linear projection based on the data from published reports, our experiments and interpolation to make this visualization.