LASER: An LLM-based ASR Scoring and Evaluation Rubric

TL;DR

LASER addresses shortcomings of WER for morphologically rich languages by using an LLM-based scoring rubric that learns from prompts with detailed examples. It combines prompt-driven in-context reasoning with a defined penalty scheme to produce sentence-level LASER scores that closely track human judgments, with cross-lingual transfer from Hindi to Marathi, Malayalam, and Kannada, where the score is computed as $Score = 1 - \frac{TotalPenalty}{N}$. The paper demonstrates strong correlations with human scores across languages, and shows that a smaller model can be trained via LoRA on word-pair data to approximate LASER scoring, enabling open-source accessibility. Overall, LASER provides a fine-grained, language-agnostic ASR evaluation approach that is scalable and better aligned with semantics than WER.

Abstract

Standard ASR evaluation metrics like Word Error Rate (WER) tend to unfairly penalize morphological and syntactic nuances that do not significantly alter sentence semantics. We introduce an LLM-based scoring rubric LASER that leverages state-of-the-art LLMs' in-context learning abilities to learn from prompts with detailed examples. Hindi LASER scores using Gemini 2.5 Pro achieved a very high correlation score of 94% with human annotations. Hindi examples in the prompt were also effective in analyzing errors in other Indian languages such as Marathi, Kannada and Malayalam. We also demonstrate how a smaller LLM like Llama 3 can be finetuned on word-pair examples derived from reference and ASR predictions to predict what kind of penalty should be applied with close to 89% accuracy.

Figures (7)

• Figure 1: Correlation heatmap for different LLM scores using the Hindi prompt, Human scores, WER and BERTScore(F1) on Hindi data.
• Figure 2: Correlation heatmap for Human scores, WER, BERTScore(F1) and Gemini 2.5 Pro scores using the Hindi (HPROM) and the Marathi (MPROM) prompts for both the Hindi and the Marathi data.
• Figure 3: Correlation heatmaps for different LLM scores using the Hindi prompt, Human scores, WER and BERTScore(F1) for 154 Marathi, 229 Malayalam and 216 Kannada sentences.
• Figure 4: Qualitative analysis of high WER samples having high and low LASER scores. Red text indicates mismatch between the original and predicted transcriptions.
• Figure 5: Correlation heatmap for finetuned Llama3, Gemini 2.5 Pro (Hindi prompt), Human, WER and BERTScore(F1) for 17 held-out Hindi sentence pairs.