Human Variability vs. Machine Consistency: A Linguistic Analysis of Texts Generated by Humans and Large Language Models

TL;DR

This study investigates how humans and large language models differ linguistically across domains by profiling texts with a comprehensive set of interpretable features. Using the M4 dataset, it combines 247 LFTK-based features with measures of syntactic depth, semantic similarity, and emotional content, applying PCA and logistic regression to distinguish human-written from machine-generated text. The results reveal that human texts exhibit greater variability, longer length, richer vocabulary, and higher emotional content, including negative emotions, while LLM outputs are more uniform and sometimes more syntactically complex. The findings highlight the importance of meaningful linguistic features for understanding and potentially detecting LLM-generated text, with implications for explainability, cross-domain analysis, and future detector design.

Abstract

The rapid advancements in large language models (LLMs) have significantly improved their ability to generate natural language, making texts generated by LLMs increasingly indistinguishable from human-written texts. Recent research has predominantly focused on using LLMs to classify text as either human-written or machine-generated. In our study, we adopt a different approach by profiling texts spanning four domains based on 250 distinct linguistic features. We select the M4 dataset from the Subtask B of SemEval 2024 Task 8. We automatically calculate various linguistic features with the LFTK tool and additionally measure the average syntactic depth, semantic similarity, and emotional content for each document. We then apply a two-dimensional PCA reduction to all the calculated features. Our analyses reveal significant differences between human-written texts and those generated by LLMs, particularly in the variability of these features, which we find to be considerably higher in human-written texts. This discrepancy is especially evident in text genres with less rigid linguistic style constraints. Our findings indicate that humans write texts that are less cognitively demanding, with higher semantic content, and richer emotional content compared to texts generated by LLMs. These insights underscore the need for incorporating meaningful linguistic features to enhance the understanding of textual outputs of LLMs.

Figures (8)

• Figure 1: Average Syntactic Depth for LLMs and Humans
• Figure 2: Semantic Distance pair-wise Sentence comparison for LLMs and Humans
• Figure 3: Average Emotion Intensity for LLMs and Humans
• Figure 4: Average Negative and Positive Emotions for LLMs and Humans
• Figure 5: Component Reduction of Text Features for LLMs and Humans