When Personalization Tricks Detectors: The Feature-Inversion Trap in Machine-Generated Text Detection

TL;DR

This work investigates how machine-generated text detectors fail when faced with personalized writing styles. It introduces StyloBench, a benchmark with Stylo-Literary and Stylo-Blog subsets to evaluate detectors under stylized domain shifts, revealing large performance drops and even inversions. The authors identify a feature-inversion trap: features that distinguish human-written from machine-generated text in general domains invert their discriminative power in personalized contexts, driving detector failures. To address this, they propose StyloCheck, a transferability estimator that uses probe datasets built around inverted features to predict detectors' post-transfer performance with high reliability (often 0.85 correlation or higher). The findings highlight the need for detection methods robust to inverted, domain-specific features and provide a practical tool for anticipating detector behavior in personalized settings, with implications for safer and more trustworthy AI systems.

Abstract

Large language models (LLMs) have grown more powerful in language generation, producing fluent text and even imitating personal style. Yet, this ability also heightens the risk of identity impersonation. To the best of our knowledge, no prior work has examined personalized machine-generated text (MGT) detection. In this paper, we introduce \dataset, the first benchmark for evaluating detector robustness in personalized settings, built from literary and blog texts paired with their LLM-generated imitations. Our experimental results demonstrate large performance gaps across detectors in personalized settings: some state-of-the-art models suffer significant drops. We attribute this limitation to the \textit{feature-inversion trap}, where features that are discriminative in general domains become inverted and misleading when applied to personalized text. Based on this finding, we propose \method, a simple and reliable way to predict detector performance changes in personalized settings. \method identifies latent directions corresponding to inverted features and constructs probe datasets that differ primarily along these features to evaluate detector dependence. Our experiments show that \method can accurately predict both the direction and the magnitude of post-transfer changes, showing 85\% correlation with the actual performance gaps. We hope that this work will encourage further research on personalized text detection.

Figures (11)

• Figure 1: Illustration of the feature-inversion trap. The feature values of HWT/MGT exhibit inverted differences across domains.
• Figure 2: AUROC scores of MGT classifiers across modules in two domains. The colors denote modules, and the line styles denote domains. We can see that the deep residual layers are best at distinguishing HWT/MGT in both domains.
• Figure 3: (a) Domain classifier score distribution: we can see moderate inversion effects. (b) Feature value distributions on the inverted feature direction $\mathbf{w}^\star$: we can see major inversion effect.
• Figure 4: Spearman $\rho$ between feature value differences of datasets and corresponding detector performance.
• Figure 5: (a) $|\mathrm{CosSim}|$ between feature directions from different datasets. $\mathbf{w}^\star$ are close to $\mathcal{W}_{\mathrm{domain}}$ and beyond $\mathcal{W}_\mathrm{MGT}$. (b) AUROC distributions of the generalization test. $\mathcal{W}_{\mathrm{domain}}$ has evidently better generalizability.