Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Continual-learning for Modelling Low-Resource Languages from Large Language Models

Santosh Srinath K, Mudit Somani, Varun Reddy Padala, Prajna Devi Upadhyay, Abhijit Das

TL;DR

This work tackles catastrophic forgetting when adapting large multilingual models to low-resource languages by introducing a parameter-efficient continual-learning framework. It uses language-specific adapters plus a shared replay adapter and a POS-guided code-switch replay mechanism to preserve previously learned cross-lingual knowledge while sequentially incorporating new languages. Experiments across MTOP, PAXQA, and xGQA demonstrate improved retention and cross-lingual generalization, with ablations validating the replay adapter and POS-based code-switch as key drivers. The approach is scalable and memory-efficient, offering a practical path for deploying multilingual VLM/LLM systems in resource-constrained settings.

Abstract

Modelling a language model for a multi-lingual scenario includes several potential challenges, among which catastrophic forgetting is the major challenge. For example, small language models (SLM) built for low-resource languages by adapting large language models (LLMs) pose the challenge of catastrophic forgetting. This work proposes to employ a continual learning strategy using parts-of-speech (POS)-based code-switching along with a replay adapter strategy to mitigate the identified gap of catastrophic forgetting while training SLM from LLM. Experiments conducted on vision language tasks such as visual question answering and language modelling task exhibits the success of the proposed architecture.

Continual-learning for Modelling Low-Resource Languages from Large Language Models

TL;DR

This work tackles catastrophic forgetting when adapting large multilingual models to low-resource languages by introducing a parameter-efficient continual-learning framework. It uses language-specific adapters plus a shared replay adapter and a POS-guided code-switch replay mechanism to preserve previously learned cross-lingual knowledge while sequentially incorporating new languages. Experiments across MTOP, PAXQA, and xGQA demonstrate improved retention and cross-lingual generalization, with ablations validating the replay adapter and POS-based code-switch as key drivers. The approach is scalable and memory-efficient, offering a practical path for deploying multilingual VLM/LLM systems in resource-constrained settings.

Abstract

Modelling a language model for a multi-lingual scenario includes several potential challenges, among which catastrophic forgetting is the major challenge. For example, small language models (SLM) built for low-resource languages by adapting large language models (LLMs) pose the challenge of catastrophic forgetting. This work proposes to employ a continual learning strategy using parts-of-speech (POS)-based code-switching along with a replay adapter strategy to mitigate the identified gap of catastrophic forgetting while training SLM from LLM. Experiments conducted on vision language tasks such as visual question answering and language modelling task exhibits the success of the proposed architecture.
Paper Structure (25 sections, 1 equation, 3 figures, 18 tables, 2 algorithms)

This paper contains 25 sections, 1 equation, 3 figures, 18 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Proposed Methodology
  3. Overview
  4. Proposed POS-Based Code Switch Replay Adapters
  5. Training procedure
  6. Experimental Setup
  7. Dataset
  8. Code-switch Lexicon Preparation
  9. Metrics
  10. Implementation details
  11. Baselines
  12. Results
  13. Comparison with Learning without Forgetting (LWF) and Elastic Weight Consolidation (EWC)
  14. Ablation Studies
  15. Effectiveness of Replay Adapter
  16. ...and 10 more sections

Figures (3)

  • Figure 1: For multilingual text tasks, the model is trained with multilingual stream (blue) using both the language adapter corresponding to each language and a shared replay adapter. During experience replay (green), only the replay adapter is updated using a POS-based code switch stream. During multimodal training (red), the image encoder functions as a visual feature extractor, while the language adapter and replay adapter jointly process textual input. The replay adapter is further updated through a POS-based code switch stream
  • Figure 2: Cross-lingual retention curve for Hindi during Bengali training in the $EN \rightarrow HI \rightarrow BN$ sequence, evaluated after each epoch to assess transfer stability
  • Figure 3: Cross-lingual retention curve for Thai during Korean training in the $EN \rightarrow TH \rightarrow KO$ sequence, evaluated after each epoch to assess transfer stability