Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Parameter-Efficient Fine-Tuning for Low-Resource Languages: A Comparative Study of LLMs for Bengali Hate Speech Detection

Akif Islam, Mohd Ruhul Ameen

TL;DR

Bengali hate speech on social media demands efficient, accessible detection methods for low-resource languages. The authors apply Parameter-Efficient Fine-Tuning (PEFT) via LoRA and QLoRA to three instruction-tuned LLMs on the BD-SHS dataset, using a single consumer GPU. Results show Llama-3.2-3B achieving 92.23% F1 on BD-SHS, with Mistral-7B and Gemma-3-4B following, all with under 1% trainable parameters and notable memory savings. This work demonstrates a practical, replicable PEFT pathway for Bengali and related Indic languages, offering deployment guidance and directions for robustness and data scaling in low-resource settings.

Abstract

Bengali social media platforms have witnessed a sharp increase in hate speech, disproportionately affecting women and adolescents. While datasets such as BD-SHS provide a basis for structured evaluation, most prior approaches rely on either computationally costly full-model fine-tuning or proprietary APIs. This paper presents the first application of Parameter-Efficient Fine-Tuning (PEFT) for Bengali hate speech detection using LoRA and QLoRA. Three instruction-tuned large language models - Gemma-3-4B, Llama-3.2-3B, and Mistral-7B - were fine-tuned on the BD-SHS dataset of 50,281 annotated comments. Each model was adapted by training fewer than 1% of its parameters, enabling experiments on a single consumer-grade GPU. The results show that Llama-3.2-3B achieved the highest F1-score of 92.23%, followed by Mistral-7B at 88.94% and Gemma-3-4B at 80.25%. These findings establish PEFT as a practical and replicable strategy for Bengali and related low-resource languages.

Parameter-Efficient Fine-Tuning for Low-Resource Languages: A Comparative Study of LLMs for Bengali Hate Speech Detection

TL;DR

Bengali hate speech on social media demands efficient, accessible detection methods for low-resource languages. The authors apply Parameter-Efficient Fine-Tuning (PEFT) via LoRA and QLoRA to three instruction-tuned LLMs on the BD-SHS dataset, using a single consumer GPU. Results show Llama-3.2-3B achieving 92.23% F1 on BD-SHS, with Mistral-7B and Gemma-3-4B following, all with under 1% trainable parameters and notable memory savings. This work demonstrates a practical, replicable PEFT pathway for Bengali and related Indic languages, offering deployment guidance and directions for robustness and data scaling in low-resource settings.

Abstract

Bengali social media platforms have witnessed a sharp increase in hate speech, disproportionately affecting women and adolescents. While datasets such as BD-SHS provide a basis for structured evaluation, most prior approaches rely on either computationally costly full-model fine-tuning or proprietary APIs. This paper presents the first application of Parameter-Efficient Fine-Tuning (PEFT) for Bengali hate speech detection using LoRA and QLoRA. Three instruction-tuned large language models - Gemma-3-4B, Llama-3.2-3B, and Mistral-7B - were fine-tuned on the BD-SHS dataset of 50,281 annotated comments. Each model was adapted by training fewer than 1% of its parameters, enabling experiments on a single consumer-grade GPU. The results show that Llama-3.2-3B achieved the highest F1-score of 92.23%, followed by Mistral-7B at 88.94% and Gemma-3-4B at 80.25%. These findings establish PEFT as a practical and replicable strategy for Bengali and related low-resource languages.
Paper Structure (21 sections, 1 equation, 3 figures, 7 tables)

This paper contains 21 sections, 1 equation, 3 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Literature Review
  3. Bengali Hate Speech Detection: From Traditional to Transformer Models
  4. Advances with Large and Tiny Language Models
  5. Parameter-Efficient Fine-Tuning Across Languages
  6. Research Gap
  7. Methodology
  8. Dataset
  9. Parameter-Efficient Fine-Tuning
  10. Full Fine-Tuning vs. PEFT
  11. LoRA and QLoRA
  12. Model Selection and Configuration
  13. Instruction Prompting
  14. Experimental Setup and Evaluation
  15. Results and Discussion
  16. ...and 6 more sections

Figures (3)

  • Figure 1: Overview of the PEFT approach. LoRA introduces trainable low-rank matrices into transformer layers, while QLoRA further reduces memory usage by quantizing base weights to 4 bits.
  • Figure 2: Confusion matrix of Llama-3.2-3B on the BD-SHS test set
  • Figure 3: Training and Validation Loss Curves for Llama-3.2-3B Fine-tuning. The curves demonstrate stable convergence without overfitting over 3 epochs, with final training loss of 0.145 and validation loss of 0.162.