Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Can We Trust LLM Detectors?

Jivnesh Sandhan, Harshit Jaiswal, Fei Cheng, Yugo Murawaki

TL;DR

The paper addresses the reliability of LLM detectors under realistic distribution shifts and unseen generators. It critiques two dominant paradigms—training-free and supervised detectors—and introduces a supervised contrastive learning framework with a DeBERTa-v3 backbone and the InfoNCE objective to learn discriminative style embeddings and enable few-shot adaptation, incorporating $L_{\mathrm{BCE}}$ and $L_{\mathrm{InfoNCE}}$. Through extensive experiments on RAID, CHEAT, and M4, it shows that while supervised detectors dominate in-domain, all methods suffer under out-of-domain conditions, and training-free methods remain brittle to proxy choices. The findings suggest that universal, domain-agnostic LLM detectors are currently infeasible, though the proposed SCL approach offers improved robustness and practical few-shot adaptation; code is available at $https://github.com/HARSHITJAIS14/DetectAI$.

Abstract

The rapid adoption of LLMs has increased the need for reliable AI text detection, yet existing detectors often fail outside controlled benchmarks. We systematically evaluate 2 dominant paradigms (training-free and supervised) and show that both are brittle under distribution shift, unseen generators, and simple stylistic perturbations. To address these limitations, we propose a supervised contrastive learning (SCL) framework that learns discriminative style embeddings. Experiments show that while supervised detectors excel in-domain, they degrade sharply out-of-domain, and training-free methods remain highly sensitive to proxy choice. Overall, our results expose fundamental challenges in building domain-agnostic detectors. Our code is available at: https://github.com/HARSHITJAIS14/DetectAI

Can We Trust LLM Detectors?

TL;DR

The paper addresses the reliability of LLM detectors under realistic distribution shifts and unseen generators. It critiques two dominant paradigms—training-free and supervised detectors—and introduces a supervised contrastive learning framework with a DeBERTa-v3 backbone and the InfoNCE objective to learn discriminative style embeddings and enable few-shot adaptation, incorporating and . Through extensive experiments on RAID, CHEAT, and M4, it shows that while supervised detectors dominate in-domain, all methods suffer under out-of-domain conditions, and training-free methods remain brittle to proxy choices. The findings suggest that universal, domain-agnostic LLM detectors are currently infeasible, though the proposed SCL approach offers improved robustness and practical few-shot adaptation; code is available at .

Abstract

The rapid adoption of LLMs has increased the need for reliable AI text detection, yet existing detectors often fail outside controlled benchmarks. We systematically evaluate 2 dominant paradigms (training-free and supervised) and show that both are brittle under distribution shift, unseen generators, and simple stylistic perturbations. To address these limitations, we propose a supervised contrastive learning (SCL) framework that learns discriminative style embeddings. Experiments show that while supervised detectors excel in-domain, they degrade sharply out-of-domain, and training-free methods remain highly sensitive to proxy choice. Overall, our results expose fundamental challenges in building domain-agnostic detectors. Our code is available at: https://github.com/HARSHITJAIS14/DetectAI
Paper Structure (12 sections, 2 equations, 2 figures, 2 tables)

This paper contains 12 sections, 2 equations, 2 figures, 2 tables.

Table of Contents

  1. Introduction
  2. The proposed framework
  3. Experiments
  4. Datasets:
  5. Baselines:
  6. Main Results:
  7. Why does our approach succeed on OOD CHEAT but fail on OOD M4?
  8. Analysis
  9. Can our framework generalize to unseen LLMs and adapt with a few examples?
  10. How robust is our proposed approach to adversarial perturbations?
  11. Related Work
  12. Conclusion

Figures (2)

  • Figure 1: Overview of the AI text detection task and proposed framework. Our framework integrates a supervised classifier with a supervised contrastive learning module that learns style embeddings, enabling robust detection.
  • Figure 2: Zero-shot vs. 25-shot detection performance of our method on unseen LLMs, showing that lightweight adaptation without retraining yields substantial gains on both GPT-4o and claude-3.5 LLMs.