Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A Two-Sample Test of Text Generation Similarity

Jingbin Xu, Chen Qian, Meimei Liu, Feng Guo

TL;DR

The paper introduces a two-sample test of text generation similarity by comparing corpus entropy estimated from neural autoregressive language models, framing document collections as generated by potentially different probability measures $\mathrm{P}_A$ and $\mathrm{P}_B$. It builds an estimation-inference framework with data-splitting to estimate entropy and conduct hypothesis testing, proving asymptotic normality under $H_0$ and addressing power loss via multiple data splits and p-value combination methods (Cauchy, MPT). Through simulations and real-data applications (AG News and NHTSA CISS crash narratives), the method maintains nominal Type I error and demonstrates superior power relative to competing approaches. The work provides a scalable, theoretically grounded tool for assessing differences in text corpora driven by underlying generative processes, with practical implications for detecting shifts in document classes and AI-generated text.

Abstract

The surge in digitized text data requires reliable inferential methods on observed textual patterns. This article proposes a novel two-sample text test for comparing similarity between two groups of documents. The hypothesis is whether the probabilistic mapping generating the textual data is identical across two groups of documents. The proposed test aims to assess text similarity by comparing the entropy of the documents. Entropy is estimated using neural network-based language models. The test statistic is derived from an estimation-and-inference framework, where the entropy is first approximated using an estimation set, followed by inference on the remaining data set. We showed theoretically that under mild conditions, the test statistic asymptotically follows a normal distribution. A multiple data-splitting strategy is proposed to enhance test power, which combines p-values into a unified decision. Various simulation studies and a real data example demonstrated that the proposed two-sample text test maintains the nominal Type one error rate while offering greater power compared to existing methods. The proposed method provides a novel solution to assert differences in document classes, particularly in fields where large-scale textual information is crucial.

A Two-Sample Test of Text Generation Similarity

TL;DR

The paper introduces a two-sample test of text generation similarity by comparing corpus entropy estimated from neural autoregressive language models, framing document collections as generated by potentially different probability measures and . It builds an estimation-inference framework with data-splitting to estimate entropy and conduct hypothesis testing, proving asymptotic normality under and addressing power loss via multiple data splits and p-value combination methods (Cauchy, MPT). Through simulations and real-data applications (AG News and NHTSA CISS crash narratives), the method maintains nominal Type I error and demonstrates superior power relative to competing approaches. The work provides a scalable, theoretically grounded tool for assessing differences in text corpora driven by underlying generative processes, with practical implications for detecting shifts in document classes and AI-generated text.

Abstract

The surge in digitized text data requires reliable inferential methods on observed textual patterns. This article proposes a novel two-sample text test for comparing similarity between two groups of documents. The hypothesis is whether the probabilistic mapping generating the textual data is identical across two groups of documents. The proposed test aims to assess text similarity by comparing the entropy of the documents. Entropy is estimated using neural network-based language models. The test statistic is derived from an estimation-and-inference framework, where the entropy is first approximated using an estimation set, followed by inference on the remaining data set. We showed theoretically that under mild conditions, the test statistic asymptotically follows a normal distribution. A multiple data-splitting strategy is proposed to enhance test power, which combines p-values into a unified decision. Various simulation studies and a real data example demonstrated that the proposed two-sample text test maintains the nominal Type one error rate while offering greater power compared to existing methods. The proposed method provides a novel solution to assert differences in document classes, particularly in fields where large-scale textual information is crucial.
Paper Structure (12 sections, 25 equations, 2 figures, 7 tables, 1 algorithm)

This paper contains 12 sections, 25 equations, 2 figures, 7 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Problem Definition
  3. Two-Sample Text Test
  4. Estimation of the entropy for a document
  5. Asymptotic behavior of the two-sample text test
  6. Data-splitting for statistical inference
  7. Theorem on power loss
  8. Numerical Results
  9. Simulation studies
  10. Benchmark dataset
  11. Real data example
  12. Discussion

Figures (2)

  • Figure 1: The random variable $D$ generated from a probability space $(\Omega, \mathcal{F}, \mathrm{P})$ based on the dictionary set $\{v_1,...v_V\}$ under different probabilistic measure mapping $\mathrm{P}$
  • Figure 2: The autoregressive neural network language model.