Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

LM$^2$otifs : An Explainable Framework for Machine-Generated Texts Detection

Xu Zheng, Zhuomin Chen, Esteban Schafir, Sipeng Chen, Hojat Allah Salehi, Haifeng Chen, Farhad Shirani, Wei Cheng, Dongsheng Luo

TL;DR

LM$^2$otifs presents an explainable framework for detecting machine-generated texts by modeling text as a co-occurrence graph of tokens and documents and applying a GNN-based detector trained on this graph. A key novelty is the post-hoc extraction of interpretable motifs via GNNExplainer, enabling explanations at word, phrase, and sentence levels grounded in probabilistic graphical model reasoning. The theoretical analysis shows that PGB detectors generalize ESB detectors and can strictly improve detection accuracy under certain distributions and context-length constraints. Comprehensive experiments across HC3, M4, RAID, and other domains demonstrate competitive performance and meaningful explainable motifs, validated with MoRF/LeRF evaluations. The approach offers a practical, interpretable alternative to black-box detectors, with implications for robust authorship authentication and forensic linguistics.

Abstract

The impressive ability of large language models to generate natural text across various tasks has led to critical challenges in authorship authentication. Although numerous detection methods have been developed to differentiate between machine-generated texts (MGT) and human-generated texts (HGT), the explainability of these methods remains a significant gap. Traditional explainability techniques often fall short in capturing the complex word relationships that distinguish HGT from MGT. To address this limitation, we present LM$^2$otifs, a novel explainable framework for MGT detection. Inspired by probabilistic graphical models, we provide a theoretical rationale for the effectiveness. LM$^2$otifs utilizes eXplainable Graph Neural Networks to achieve both accurate detection and interpretability. The LM$^2$otifs pipeline operates in three key stages: first, it transforms text into graphs based on word co-occurrence to represent lexical dependencies; second, graph neural networks are used for prediction; and third, a post-hoc explainability method extracts interpretable motifs, offering multi-level explanations from individual words to sentence structures. Extensive experiments on multiple benchmark datasets demonstrate the comparable performance of LM$^2$otifs. The empirical evaluation of the extracted explainable motifs confirms their effectiveness in differentiating HGT and MGT. Furthermore, qualitative analysis reveals distinct and visible linguistic fingerprints characteristic of MGT.

LM$^2$otifs : An Explainable Framework for Machine-Generated Texts Detection

TL;DR

LMotifs presents an explainable framework for detecting machine-generated texts by modeling text as a co-occurrence graph of tokens and documents and applying a GNN-based detector trained on this graph. A key novelty is the post-hoc extraction of interpretable motifs via GNNExplainer, enabling explanations at word, phrase, and sentence levels grounded in probabilistic graphical model reasoning. The theoretical analysis shows that PGB detectors generalize ESB detectors and can strictly improve detection accuracy under certain distributions and context-length constraints. Comprehensive experiments across HC3, M4, RAID, and other domains demonstrate competitive performance and meaningful explainable motifs, validated with MoRF/LeRF evaluations. The approach offers a practical, interpretable alternative to black-box detectors, with implications for robust authorship authentication and forensic linguistics.

Abstract

The impressive ability of large language models to generate natural text across various tasks has led to critical challenges in authorship authentication. Although numerous detection methods have been developed to differentiate between machine-generated texts (MGT) and human-generated texts (HGT), the explainability of these methods remains a significant gap. Traditional explainability techniques often fall short in capturing the complex word relationships that distinguish HGT from MGT. To address this limitation, we present LMotifs, a novel explainable framework for MGT detection. Inspired by probabilistic graphical models, we provide a theoretical rationale for the effectiveness. LMotifs utilizes eXplainable Graph Neural Networks to achieve both accurate detection and interpretability. The LMotifs pipeline operates in three key stages: first, it transforms text into graphs based on word co-occurrence to represent lexical dependencies; second, graph neural networks are used for prediction; and third, a post-hoc explainability method extracts interpretable motifs, offering multi-level explanations from individual words to sentence structures. Extensive experiments on multiple benchmark datasets demonstrate the comparable performance of LMotifs. The empirical evaluation of the extracted explainable motifs confirms their effectiveness in differentiating HGT and MGT. Furthermore, qualitative analysis reveals distinct and visible linguistic fingerprints characteristic of MGT.
Paper Structure (22 sections, 1 theorem, 13 equations, 6 figures, 24 tables)

This paper contains 22 sections, 1 theorem, 13 equations, 6 figures, 24 tables.

Table of Contents

  1. Introduction
  2. Preliminary
  3. Methodology
  4. Graph Construction
  5. GNN Detection
  6. Explainable Motifs Extraction
  7. Theoretical Analysis
  8. Related Work
  9. AI-generated text Detection
  10. Explainable LLMs & GNNs
  11. Experiments
  12. Setups
  13. Detection Performance Comparison
  14. Explanation Evaluation
  15. Conclusion
  16. ...and 7 more sections

Key Result

Theorem 4.1

For every ESB detector $f_{\text{ESB}} \in \mathcal{F}_{\text{ESB}}$, there exists a PGB detector $f_{\text{PGB}} \in \mathcal{F}_{\text{PGB}}$ such that the detection accuracy of $f_{\text{PGB}}$ matches that of $f_{\text{ESB}}$, i.e., for all pairs of probability distributions $(P_h,P_m)$. Furthermore, the PGB class of detectors strictly improves upon the ESB class in terms of detection accurac

Figures (6)

  • Figure 1: A PGM example of a three-token text.
  • Figure 2: Overall pipeline of our framework, including tokenization, graph building, detector training, and motifs extraction.
  • Figure 3: An example of graph construction with a fixed slid window 3.
  • Figure 4: Comparison results of MoRF and LeRF between explainable motifs extracted from LM$^2$otifs and random motifs.
  • Figure 5: High-order explainable motif samples from GPT-4 and Davinci. We extract motifs from texts in the PubMed dataset for the same question. In graph motifs, solid lines represent subgraph motifs and dashed lines mean the text contains words. In text motifs, words highlighted in the same color are connected in the corresponding graph motifs. A single word may contain multiple colors.
  • ...and 1 more figures

Theorems & Definitions (1)

  • Theorem 4.1