Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

RFBES at SemEval-2024 Task 8: Investigating Syntactic and Semantic Features for Distinguishing AI-Generated and Human-Written Texts

Mohammad Heydari Rad, Farhan Farsi, Shayan Bali, Romina Etezadi, Mehrnoush Shamsfard

TL;DR

The paper tackles AI-generated text detection across languages using the M4 dataset. It develops two analysis tracks: a semantic pathway based on segmentation and a transformer classifier, and a syntactic pathway based on UPOS sequences processed by a BiLSTM with attention. On multilingual tasks it achieves $0.847$ accuracy and $0.859$ on monolingual tasks, with semantic analysis providing the majority of gains and syntactic signals being largely non-discriminative. Semantic-driven, small-segment analysis improves performance by $3.9\%$ over the official multilingual baseline, and the authors propose Graph Neural Networks to exploit grammatical connections. This work highlights the importance of semantic cues in AI text detection and provides a framework for integrating syntactic signals, with potential applications in content moderation and AI transparency.

Abstract

Nowadays, the usage of Large Language Models (LLMs) has increased, and LLMs have been used to generate texts in different languages and for different tasks. Additionally, due to the participation of remarkable companies such as Google and OpenAI, LLMs are now more accessible, and people can easily use them. However, an important issue is how we can detect AI-generated texts from human-written ones. In this article, we have investigated the problem of AI-generated text detection from two different aspects: semantics and syntax. Finally, we presented an AI model that can distinguish AI-generated texts from human-written ones with high accuracy on both multilingual and monolingual tasks using the M4 dataset. According to our results, using a semantic approach would be more helpful for detection. However, there is a lot of room for improvement in the syntactic approach, and it would be a good approach for future work.

RFBES at SemEval-2024 Task 8: Investigating Syntactic and Semantic Features for Distinguishing AI-Generated and Human-Written Texts

TL;DR

The paper tackles AI-generated text detection across languages using the M4 dataset. It develops two analysis tracks: a semantic pathway based on segmentation and a transformer classifier, and a syntactic pathway based on UPOS sequences processed by a BiLSTM with attention. On multilingual tasks it achieves accuracy and on monolingual tasks, with semantic analysis providing the majority of gains and syntactic signals being largely non-discriminative. Semantic-driven, small-segment analysis improves performance by over the official multilingual baseline, and the authors propose Graph Neural Networks to exploit grammatical connections. This work highlights the importance of semantic cues in AI text detection and provides a framework for integrating syntactic signals, with potential applications in content moderation and AI transparency.

Abstract

Nowadays, the usage of Large Language Models (LLMs) has increased, and LLMs have been used to generate texts in different languages and for different tasks. Additionally, due to the participation of remarkable companies such as Google and OpenAI, LLMs are now more accessible, and people can easily use them. However, an important issue is how we can detect AI-generated texts from human-written ones. In this article, we have investigated the problem of AI-generated text detection from two different aspects: semantics and syntax. Finally, we presented an AI model that can distinguish AI-generated texts from human-written ones with high accuracy on both multilingual and monolingual tasks using the M4 dataset. According to our results, using a semantic approach would be more helpful for detection. However, there is a lot of room for improvement in the syntactic approach, and it would be a good approach for future work.
Paper Structure (7 sections, 3 figures, 1 table)

This paper contains 7 sections, 3 figures, 1 table.

Table of Contents

  1. Introduction
  2. Background
  3. Method
  4. Semantic Approach
  5. Syntactic Approach
  6. Result
  7. Conclusion

Figures (3)

  • Figure 1: The input text is divided into meaningful units, and the probability of each segment based on their logits is assessed using a fine-tuned XLM-RoBERTa model; the combined evidence leads to a definitive classification.
  • Figure 2: The bidirectional LSTM model predicts using part-of-speech labels associated with the words in the text assessed by Trankit.
  • Figure 3: Confusion matrices of our model for test datasets on monolingual and multilingual tracks