Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Granary: Speech Recognition and Translation Dataset in 25 European Languages

Nithin Rao Koluguri, Monica Sekoyan, George Zelenfroynd, Sasha Meister, Shuoyang Ding, Sofia Kostandian, He Huang, Nikolay Karpov, Jagadeesh Balam, Vitaly Lavrukhin, Yifan Peng, Sara Papi, Marco Gaido, Alessio Brutti, Boris Ginsburg

TL;DR

Granary tackles the data scarcity problem in multilingual speech by introducing an open-source, end-to-end pseudo-labeling pipeline for ASR and AST across 25 European languages. The approach combines long-form audio segmentation, two-pass inference, LID verification, robust filtration, and LLM-based punctuation restoration to generate high-quality pseudo-labeled data at scale, then uses EuroLLM for translation pseudo-labeling with a NeMo-Curator filtration step. The study processes roughly 1 million hours of unlabeled data to produce about 638k hours for ASR and 351k hours for AST, achieving strong retention (≈60%) and enabling performance on par with larger, manually labeled datasets using roughly half the data. Evaluation on English and Croatian shows Granary-filtered data yields comparable or improved WER relative to MOSEL, with notable gains on FLEURS and low-resource settings, highlighting the practical impact for building open, multilingual speech models. Future work aims to release multi-task, multilingual models trained on the full Granary corpus, further broadening accessibility and impact.

Abstract

Multi-task and multilingual approaches benefit large models, yet speech processing for low-resource languages remains underexplored due to data scarcity. To address this, we present Granary, a large-scale collection of speech datasets for recognition and translation across 25 European languages. This is the first open-source effort at this scale for both transcription and translation. We enhance data quality using a pseudo-labeling pipeline with segmentation, two-pass inference, hallucination filtering, and punctuation restoration. We further generate translation pairs from pseudo-labeled transcriptions using EuroLLM, followed by a data filtration pipeline. Designed for efficiency, our pipeline processes vast amount of data within hours. We assess models trained on processed data by comparing their performance on previously curated datasets for both high- and low-resource languages. Our findings show that these models achieve similar performance using approx. 50% less data. Dataset will be made available at https://hf.co/datasets/nvidia/Granary

Granary: Speech Recognition and Translation Dataset in 25 European Languages

TL;DR

Granary tackles the data scarcity problem in multilingual speech by introducing an open-source, end-to-end pseudo-labeling pipeline for ASR and AST across 25 European languages. The approach combines long-form audio segmentation, two-pass inference, LID verification, robust filtration, and LLM-based punctuation restoration to generate high-quality pseudo-labeled data at scale, then uses EuroLLM for translation pseudo-labeling with a NeMo-Curator filtration step. The study processes roughly 1 million hours of unlabeled data to produce about 638k hours for ASR and 351k hours for AST, achieving strong retention (≈60%) and enabling performance on par with larger, manually labeled datasets using roughly half the data. Evaluation on English and Croatian shows Granary-filtered data yields comparable or improved WER relative to MOSEL, with notable gains on FLEURS and low-resource settings, highlighting the practical impact for building open, multilingual speech models. Future work aims to release multi-task, multilingual models trained on the full Granary corpus, further broadening accessibility and impact.

Abstract

Multi-task and multilingual approaches benefit large models, yet speech processing for low-resource languages remains underexplored due to data scarcity. To address this, we present Granary, a large-scale collection of speech datasets for recognition and translation across 25 European languages. This is the first open-source effort at this scale for both transcription and translation. We enhance data quality using a pseudo-labeling pipeline with segmentation, two-pass inference, hallucination filtering, and punctuation restoration. We further generate translation pairs from pseudo-labeled transcriptions using EuroLLM, followed by a data filtration pipeline. Designed for efficiency, our pipeline processes vast amount of data within hours. We assess models trained on processed data by comparing their performance on previously curated datasets for both high- and low-resource languages. Our findings show that these models achieve similar performance using approx. 50% less data. Dataset will be made available at https://hf.co/datasets/nvidia/Granary
Paper Structure (15 sections, 3 figures, 3 tables)

This paper contains 15 sections, 3 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Data
  3. Granary Pipeline
  4. ASR Data Pipeline
  5. Long-form Audio Segmentation
  6. Two-Pass Inference
  7. LID Verification
  8. Robust Data Filtration
  9. LLM-Powered P&C Restoration
  10. AST Data Pipeline
  11. Selection of Pseudo-Labeling Models
  12. LLM Inference
  13. Filtration
  14. Model Training and Evaluation
  15. Conclusion

Figures (3)

  • Figure 1: Granary pseudo-labeling pipeline. The pipeline consists of two parts: ASR and AST. The ASR pipeline includes segmentation, two-pass ASR model inference, language ID verification, text filtering, and Punctuation and Capitalization (PnC) restoration. The AST pipeline involves AST pair generation using the EuroLLM model, followed by Quality Estimation filtering.
  • Figure 2: Example of MOSEL Transcription Truncation Issue and the same sentence transcription with Granary Pipeline.
  • Figure 3: Total hours per language per task (ASR and AST) in Granary after final filtering (log scale). English has the most hours (275k for ASR), while Ukrainian has the least (932.67 for ASR, 608.80 for X$\rightarrow$En).