Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

BhashaSetu: Cross-Lingual Knowledge Transfer from High-Resource to Extreme Low-Resource Languages

Subhadip Maji, Arnab Bhattacharya

TL;DR

This paper tackles cross-lingual knowledge transfer for extreme low-resource languages by introducing BhashaSetu, a framework that combines Hidden Augmentation Layers (HAL), Token Embedding Transfer (TET), and a novel Graph-Enhanced Token Representation (GETR) method. GETR integrates Graph Neural Networks into transformer architectures to enable dynamic, token-level cross-lingual sharing, augmented by a strategic batch formation and cross-script edge construction. Empirical results across sentiment classification, NER, and POS tagging show substantial macro-F1 gains for truly and simulated low-resource languages, with gains up to 13–27 percentage points and modest computational overhead. The findings demonstrate significant potential for data-efficient cross-lingual transfer to hundreds of under-resourced languages, particularly in multilingual contexts like Indian language families.

Abstract

Despite remarkable advances in natural language processing, developing effective systems for low-resource languages remains a formidable challenge, with performances typically lagging far behind high-resource counterparts due to data scarcity and insufficient linguistic resources. Cross-lingual knowledge transfer has emerged as a promising approach to address this challenge by leveraging resources from high-resource languages. In this paper, we investigate methods for transferring linguistic knowledge from high-resource languages to low-resource languages, where the number of labeled training instances is in hundreds. We focus on sentence-level and word-level tasks. We introduce a novel method, GETR (Graph-Enhanced Token Representation) for cross-lingual knowledge transfer along with two adopted baselines (a) augmentation in hidden layers and (b) token embedding transfer through token translation. Experimental results demonstrate that our GNN-based approach significantly outperforms existing multilingual and cross-lingual baseline methods, achieving 13 percentage point improvements on truly low-resource languages (Mizo, Khasi) for POS tagging, and 20 and 27 percentage point improvements in macro-F1 on simulated low-resource languages (Marathi, Bangla, Malayalam) across sentiment classification and NER tasks respectively. We also present a detailed analysis of the transfer mechanisms and identify key factors that contribute to successful knowledge transfer in this linguistic context.

BhashaSetu: Cross-Lingual Knowledge Transfer from High-Resource to Extreme Low-Resource Languages

TL;DR

This paper tackles cross-lingual knowledge transfer for extreme low-resource languages by introducing BhashaSetu, a framework that combines Hidden Augmentation Layers (HAL), Token Embedding Transfer (TET), and a novel Graph-Enhanced Token Representation (GETR) method. GETR integrates Graph Neural Networks into transformer architectures to enable dynamic, token-level cross-lingual sharing, augmented by a strategic batch formation and cross-script edge construction. Empirical results across sentiment classification, NER, and POS tagging show substantial macro-F1 gains for truly and simulated low-resource languages, with gains up to 13–27 percentage points and modest computational overhead. The findings demonstrate significant potential for data-efficient cross-lingual transfer to hundreds of under-resourced languages, particularly in multilingual contexts like Indian language families.

Abstract

Despite remarkable advances in natural language processing, developing effective systems for low-resource languages remains a formidable challenge, with performances typically lagging far behind high-resource counterparts due to data scarcity and insufficient linguistic resources. Cross-lingual knowledge transfer has emerged as a promising approach to address this challenge by leveraging resources from high-resource languages. In this paper, we investigate methods for transferring linguistic knowledge from high-resource languages to low-resource languages, where the number of labeled training instances is in hundreds. We focus on sentence-level and word-level tasks. We introduce a novel method, GETR (Graph-Enhanced Token Representation) for cross-lingual knowledge transfer along with two adopted baselines (a) augmentation in hidden layers and (b) token embedding transfer through token translation. Experimental results demonstrate that our GNN-based approach significantly outperforms existing multilingual and cross-lingual baseline methods, achieving 13 percentage point improvements on truly low-resource languages (Mizo, Khasi) for POS tagging, and 20 and 27 percentage point improvements in macro-F1 on simulated low-resource languages (Marathi, Bangla, Malayalam) across sentiment classification and NER tasks respectively. We also present a detailed analysis of the transfer mechanisms and identify key factors that contribute to successful knowledge transfer in this linguistic context.
Paper Structure (26 sections, 4 equations, 3 figures, 14 tables, 1 algorithm)

This paper contains 26 sections, 4 equations, 3 figures, 14 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Methodology
  4. Problem Statement:
  5. HAL
  6. TET
  7. GETR
  8. Strategic Batch Formation for Graph Construction:
  9. Cross-Script Edge Construction:
  10. Experiments and Results
  11. Dataset
  12. Implementation Details
  13. Results on Sentiment Classification Task
  14. Results on NER Task
  15. Results on POS Tagging Task
  16. ...and 11 more sections

Figures (3)

  • Figure 1: Graphical representation of tokens of two sentences in a batch: "The movie was good" and "I was impressed with the movie".
  • Figure 2: BERT encoder architecture incorporating the GNN layer for cross-lingual knowledge transfer.
  • Figure 3: Architecture incorporating the Hidden Augmentation Layer (HRL and LRL inputs are high- and low-resource language inputs respectively)