Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Detecting AI-Generated Paraphrases in Bengali: A Comparative Study of Zero-Shot and Fine-Tuned Transformers

Md. Rakibul Islam, Most. Sharmin Sultana Samu, Md. Zahid Hossain, Farhad Uz Zaman, Md. Kamrozzaman Bhuiyan

TL;DR

This study addresses the challenge of detecting AI-generated Bengali paraphrases by benchmarking five transformer models in zero-shot and fine-tuned settings on the BanglaTextDistinguish dataset. Zero-shot results are near random, highlighting the need for task-specific training. Fine-tuned models achieve high accuracy (~91%) and strong calibration, with XLM-RoBERTa-Large and mDeBERTaV3-Base leading, while IndicBERT lags. Compared to traditional baselines, transformer approaches substantially improve detection robustness, underscoring their potential for real-world Bengali AI-content countermeasures and identifying avenues for future improvement.

Abstract

Large language models (LLMs) can produce text that closely resembles human writing. This capability raises concerns about misuse, including disinformation and content manipulation. Detecting AI-generated text is essential to maintain authenticity and prevent malicious applications. Existing research has addressed detection in multiple languages, but the Bengali language remains largely unexplored. Bengali's rich vocabulary and complex structure make distinguishing human-written and AI-generated text particularly challenging. This study investigates five transformer-based models: XLMRoBERTa-Large, mDeBERTaV3-Base, BanglaBERT-Base, IndicBERT-Base and MultilingualBERT-Base. Zero-shot evaluation shows that all models perform near chance levels (around 50% accuracy) and highlight the need for task-specific fine-tuning. Fine-tuning significantly improves performance, with XLM-RoBERTa, mDeBERTa and MultilingualBERT achieving around 91% on both accuracy and F1-score. IndicBERT demonstrates comparatively weaker performance, indicating limited effectiveness in fine-tuning for this task. This work advances AI-generated text detection in Bengali and establishes a foundation for building robust systems to counter AI-generated content.

Detecting AI-Generated Paraphrases in Bengali: A Comparative Study of Zero-Shot and Fine-Tuned Transformers

TL;DR

This study addresses the challenge of detecting AI-generated Bengali paraphrases by benchmarking five transformer models in zero-shot and fine-tuned settings on the BanglaTextDistinguish dataset. Zero-shot results are near random, highlighting the need for task-specific training. Fine-tuned models achieve high accuracy (~91%) and strong calibration, with XLM-RoBERTa-Large and mDeBERTaV3-Base leading, while IndicBERT lags. Compared to traditional baselines, transformer approaches substantially improve detection robustness, underscoring their potential for real-world Bengali AI-content countermeasures and identifying avenues for future improvement.

Abstract

Large language models (LLMs) can produce text that closely resembles human writing. This capability raises concerns about misuse, including disinformation and content manipulation. Detecting AI-generated text is essential to maintain authenticity and prevent malicious applications. Existing research has addressed detection in multiple languages, but the Bengali language remains largely unexplored. Bengali's rich vocabulary and complex structure make distinguishing human-written and AI-generated text particularly challenging. This study investigates five transformer-based models: XLMRoBERTa-Large, mDeBERTaV3-Base, BanglaBERT-Base, IndicBERT-Base and MultilingualBERT-Base. Zero-shot evaluation shows that all models perform near chance levels (around 50% accuracy) and highlight the need for task-specific fine-tuning. Fine-tuning significantly improves performance, with XLM-RoBERTa, mDeBERTa and MultilingualBERT achieving around 91% on both accuracy and F1-score. IndicBERT demonstrates comparatively weaker performance, indicating limited effectiveness in fine-tuning for this task. This work advances AI-generated text detection in Bengali and establishes a foundation for building robust systems to counter AI-generated content.
Paper Structure (15 sections, 5 figures, 6 tables)

This paper contains 15 sections, 5 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Background Study
  4. Transformer Models for AI-Paraphrased Text Classification
  5. Evaluation Metrics
  6. Methodology
  7. Dataset and Preprocessing
  8. Zero-Shot Inference Models
  9. Fine-Tuned Models
  10. Experimental Setup
  11. Result Analysis
  12. Zero-Shot Classification
  13. Fine-Tuned Classification
  14. Comparison With Existing Work
  15. Conclusion and Future Works

Figures (5)

  • Figure 1: Proposed Methodology
  • Figure 2: Confusion matrices of the fine-tuned classification models.
  • Figure 3: ROC curve of the fine-tuned classification models.
  • Figure 4: DET curve of the fine-tuned classification models.
  • Figure 5: Reliability (Calibration) curve of the fine-tuned classification models.