Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

AI Detectors Fail Diverse Student Populations: A Mathematical Framing of Structural Detection Limits

Nathan Garland

Abstract

Student experiences and empirical studies report that "black box" AI text detectors produce high false positive rates with disproportionate errors against certain student populations, yet typically theoretical analyses model detection as a test between two known distributions for human and AI prose. This framing omits the structural feature of university assessment whereby an assessor generally does not know the individual student's writing distribution, making the null hypothesis composite. Standard application of the variational characterisation of total variation distance to this composite null shows trade-off bounds that any text-only, one-shot detector with useful power must produce false accusations at a rate governed by the distributional overlap between student writing and AI output. This is a constraint arising from population diversity that is logically independent of AI model quality and cannot be overcome by better detector engineering or technology. A subgroup mixture bound connects these quantities to observable demographic groups, providing a theoretical basis for the disparate impact patterns documented empirically. We propose suggestions to improve policy and practice, and argue that detection scores should not serve as sole evidence in misconduct proceedings.

AI Detectors Fail Diverse Student Populations: A Mathematical Framing of Structural Detection Limits

Abstract

Student experiences and empirical studies report that "black box" AI text detectors produce high false positive rates with disproportionate errors against certain student populations, yet typically theoretical analyses model detection as a test between two known distributions for human and AI prose. This framing omits the structural feature of university assessment whereby an assessor generally does not know the individual student's writing distribution, making the null hypothesis composite. Standard application of the variational characterisation of total variation distance to this composite null shows trade-off bounds that any text-only, one-shot detector with useful power must produce false accusations at a rate governed by the distributional overlap between student writing and AI output. This is a constraint arising from population diversity that is logically independent of AI model quality and cannot be overcome by better detector engineering or technology. A subgroup mixture bound connects these quantities to observable demographic groups, providing a theoretical basis for the disparate impact patterns documented empirically. We propose suggestions to improve policy and practice, and argue that detection scores should not serve as sole evidence in misconduct proceedings.
Paper Structure (18 sections, 3 theorems, 11 equations, 2 figures)

This paper contains 18 sections, 3 theorems, 11 equations, 2 figures.

Table of Contents

  1. Introduction
  2. Background
  3. Methods
  4. Setting the University Detection Problem
  5. Composite Null
  6. Theoretical Results Any Detector Must Face
  7. An Average-Case Trade-Off
  8. Worst-Case Protection
  9. Subgroup Mixture Bound
  10. Discussion
  11. Two Distinct Sources of Detection Difficulty
  12. Connecting Theory to Practice: Estimation and Auditing
  13. Task Dependence
  14. Limitations
  15. A Practical Path: Direct FPR Auditing
  16. ...and 3 more sections

Key Result

Theorem 1

Let $\{p_\theta\}_{\theta \in \Theta}$ be a family of distributions, $p_M$ a fixed distribution, and $\pi$ a probability measure on $\Theta$. For $\delta > 0$, let $\Theta^* = \{\theta : \mathrm{TV}(p_\theta, p_M) \leq \delta\}$ with $\pi(\Theta^*) > 0$. Then for any $\phi: \mathcal{X} \to [0,1]$:

Figures (2)

  • Figure 1: Schematic comparison of the testing structure. Left: prior theoretical work assumes a single known human distribution $p_H$ and single known AI distribution $p_M$. Right: the university setting involves a composite null---each student has a different writing distribution $p_\theta$, and the detector does not know which $\theta$ applies. Some students (e.g. those where English is a second language, or students writing a discipline specific assessment that uses structured norms) may have distributions very close to $p_M$ in total variation distance.
  • Figure 2: The lower bound on population-averaged FPR from Result 1 at detector power $\beta_0 = 0.80$, shown across the $(\pi(\Theta^*(\delta)), \delta)$ parameter space. Note that $\pi(\Theta^*(\delta))$ is non-decreasing in $\delta$ by construction, so a given student population traces a monotone curve through this space rather than occupying it freely. The figure shows the bound value at each mathematically valid parameter pair; the realised pair for any specific population and task is an empirical question. The illustrative example from Section 3.3.1 is marked.

Theorems & Definitions (7)

  • Theorem 1: Average-case trade-off
  • proof
  • Theorem 2: Worst-case size--power bound
  • proof
  • Theorem 3: Subgroup mixture bound
  • proof
  • Remark 1: Convexity