Transformer and Hybrid Deep Learning Based Models for Machine-Generated Text Detection

TL;DR

The paper addresses multilingual, multi-domain detection of machine-generated text in SemEval 2024 Task 8 by evaluating transformer-based detectors and a CNN+BiLSTM hybrid for token-level classification. A two-phase training regime (freeze then selective fine-tuning) and careful layer selection are used to mitigate overfitting, while a long-text handling strategy leverages head-only truncation to 512 tokens. The transformer-based approach achieves a strong second-place result in subtask B (86.95% accuracy), but overfits in subtask A and subtask C, highlighting limitations in handling longer contexts and token-level transitions. The authors propose extending sequence lengths, ensembling with model specialization, and exploring LLM-based few-shot or zero-shot approaches to improve robustness and generalization in future work.

Abstract

This paper describes the approach of the UniBuc - NLP team in tackling the SemEval 2024 Task 8: Multigenerator, Multidomain, and Multilingual Black-Box Machine-Generated Text Detection. We explored transformer-based and hybrid deep learning architectures. For subtask B, our transformer-based model achieved a strong \textbf{second-place} out of $77$ teams with an accuracy of \textbf{86.95\%}, demonstrating the architecture's suitability for this task. However, our models showed overfitting in subtask A which could potentially be fixed with less fine-tunning and increasing maximum sequence length. For subtask C (token-level classification), our hybrid model overfit during training, hindering its ability to detect transitions between human and machine-generated text.

Figures (9)

• Figure 1: Three sub-tasks explained
• Figure 2: Fully connected layer base structure
• Figure 3: Transformer based models architecture
• Figure 4: CNN-character level features
• Figure 5: Hybrid deep learning model architectures. Method 1 to use the predictions directly from the fully connected block and method 2 using CRF before predictions.