MAGE: Multi-Head Attention Guided Embeddings for Low Resource Sentiment Classification

TL;DR

The paper tackles sentiment classification in low-resource Bantu languages where labeled data are scarce. It extends Language-Independent Data Augmentation (LiDA) by integrating AfriBERTa embeddings, replacing the denoising autoencoder with a variational autoencoder, and applying a Multi-Head Attention-based fusion to weight embeddings, yielding richer, more discriminative representations. Evaluations on AfriSenti SemEval data for Kinyarwanda, Xitsonga, and Swahili demonstrate that MAGE outperforms LiDA baselines, with improvements up to $3.64\%$ in accuracy when using DAE and $2.51\%$ with VAE, along with additional gains from attention. The framework reduces reliance on language-specific resources and provides a scalable approach for expanding NLP capabilities to other low-resource language families.

Abstract

Due to the lack of quality data for low-resource Bantu languages, significant challenges are presented in text classification and other practical implementations. In this paper, we introduce an advanced model combining Language-Independent Data Augmentation (LiDA) with Multi-Head Attention based weighted embeddings to selectively enhance critical data points and improve text classification performance. This integration allows us to create robust data augmentation strategies that are effective across various linguistic contexts, ensuring that our model can handle the unique syntactic and semantic features of Bantu languages. This approach not only addresses the data scarcity issue but also sets a foundation for future research in low-resource language processing and classification tasks.

Figures (4)

• Figure 1: LiDA Framework Reproduced from LiDALiDA.
• Figure 2: Modified LiDA - MAGE Framework
• Figure 3: Comparative Results
• Figure 4: Comparative Results