Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

FUSE : A Ridge and Random Forest-Based Metric for Evaluating MT in Indigenous Languages

Rahul Raja, Arpita Vats

TL;DR

This work tackles automatic MT evaluation for morphologically rich Indigenous languages by introducing FUSE, a supervised metric that fuses lexical, phonetic, semantic, and fuzzy similarity signals. It explores six increasingly sophisticated approaches, culminating in a Ridge–Random Forest ensemble and a Ridge–Gradient Boosting ensemble that learn language-specific weights from human scores. Across Bribri, Guarani, and Nahuatl, FUSE-based methods yield higher Pearson and Spearman correlations with human judgments than traditional metrics, demonstrating improved alignment with semantic adequacy and fluency in low-resource settings. The results highlight the viability of linguistically informed, data-driven evaluation for morphologically complex languages and suggest broad applicability to other low-resource MT evaluation tasks.

Abstract

This paper presents the winning submission of the RaaVa team to the AmericasNLP 2025 Shared Task 3 on Automatic Evaluation Metrics for Machine Translation (MT) into Indigenous Languages of America, where our system ranked first overall based on average Pearson correlation with the human annotations. We introduce Feature-Union Scorer (FUSE) for Evaluation, FUSE integrates Ridge regression and Gradient Boosting to model translation quality. In addition to FUSE, we explore five alternative approaches leveraging different combinations of linguistic similarity features and learning paradigms. FUSE Score highlights the effectiveness of combining lexical, phonetic, semantic, and fuzzy token similarity with learning-based modeling to improve MT evaluation for morphologically rich and low-resource languages. MT into Indigenous languages poses unique challenges due to polysynthesis, complex morphology, and non-standardized orthography. Conventional automatic metrics such as BLEU, TER, and ChrF often fail to capture deeper aspects like semantic adequacy and fluency. Our proposed framework, formerly referred to as FUSE, incorporates multilingual sentence embeddings and phonological encodings to better align with human evaluation. We train supervised models on human-annotated development sets and evaluate held-out test data. Results show that FUSE consistently achieves higher Pearson and Spearman correlations with human judgments, offering a robust and linguistically informed solution for MT evaluation in low-resource settings.

FUSE : A Ridge and Random Forest-Based Metric for Evaluating MT in Indigenous Languages

TL;DR

This work tackles automatic MT evaluation for morphologically rich Indigenous languages by introducing FUSE, a supervised metric that fuses lexical, phonetic, semantic, and fuzzy similarity signals. It explores six increasingly sophisticated approaches, culminating in a Ridge–Random Forest ensemble and a Ridge–Gradient Boosting ensemble that learn language-specific weights from human scores. Across Bribri, Guarani, and Nahuatl, FUSE-based methods yield higher Pearson and Spearman correlations with human judgments than traditional metrics, demonstrating improved alignment with semantic adequacy and fluency in low-resource settings. The results highlight the viability of linguistically informed, data-driven evaluation for morphologically complex languages and suggest broad applicability to other low-resource MT evaluation tasks.

Abstract

This paper presents the winning submission of the RaaVa team to the AmericasNLP 2025 Shared Task 3 on Automatic Evaluation Metrics for Machine Translation (MT) into Indigenous Languages of America, where our system ranked first overall based on average Pearson correlation with the human annotations. We introduce Feature-Union Scorer (FUSE) for Evaluation, FUSE integrates Ridge regression and Gradient Boosting to model translation quality. In addition to FUSE, we explore five alternative approaches leveraging different combinations of linguistic similarity features and learning paradigms. FUSE Score highlights the effectiveness of combining lexical, phonetic, semantic, and fuzzy token similarity with learning-based modeling to improve MT evaluation for morphologically rich and low-resource languages. MT into Indigenous languages poses unique challenges due to polysynthesis, complex morphology, and non-standardized orthography. Conventional automatic metrics such as BLEU, TER, and ChrF often fail to capture deeper aspects like semantic adequacy and fluency. Our proposed framework, formerly referred to as FUSE, incorporates multilingual sentence embeddings and phonological encodings to better align with human evaluation. We train supervised models on human-annotated development sets and evaluate held-out test data. Results show that FUSE consistently achieves higher Pearson and Spearman correlations with human judgments, offering a robust and linguistically informed solution for MT evaluation in low-resource settings.

Paper Structure

This paper contains 18 sections, 6 equations, 1 figure, 3 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Rule-Based Metrics
  4. Embedding-Based Metrics
  5. Learning-Based Metrics
  6. Quality Estimation (Reference-Free Metrics)
  7. Datasets
  8. Proposed Methods
  9. Approach 1: Lexical and Phonetic Baseline
  10. Approach 2: Feature-Enriched Similarity (DistilUSE)
  11. Approach 3: Weighted Similarity Aggregation
  12. Approach 4: Data-Driven Weighted Similarity via Regression
  13. Approach 5: Hybrid Regression with Ridge and Random Forest
  14. Approach 6: Ensemble Regression with Ridge and Gradient Boosting
  15. Implementation Details
  16. ...and 3 more sections

Figures (1)

  • Figure 1: FUSE architecture combining linguistic features with hybrid regression for MT evaluation.