Counterfactual LLM-based Framework for Measuring Rhetorical Style

TL;DR

A counterfactual, LLM-based framework that generates more than 250,000 counterfactual writings and provides a large-scale quantification of rhetorical style in ML papers, and demonstrates that LLMs can serve as instruments to measure and improve scientific evaluation.

Abstract

The rise of AI has fueled growing concerns about ``hype'' in machine learning papers, yet a reliable way to quantify rhetorical style independently of substantive content has remained elusive. Because bold language can stem from either strong empirical results or mere rhetorical style, it is often difficult to distinguish between the two. To disentangle rhetorical style from substantive content, we introduce a counterfactual, LLM-based framework: multiple LLM rhetorical personas generate counterfactual writings from the same substantive content, an LLM judge compares them through pairwise evaluations, and the outcomes are aggregated using a Bradley--Terry model. Applying this method to 8,485 ICLR submissions sampled from 2017 to 2025, we generate more than 250,000 counterfactual writings and provide a large-scale quantification of rhetorical style in ML papers. We find that visionary framing significantly predicts downstream attention, including citations and media attention, even after controlling for peer-review evaluations. We also observe a sharp rise in rhetorical strength after 2023, and provide empirical evidence showing that this increase is largely driven by the adoption of LLM-based writing assistance. The reliability of our framework is validated by its robustness to the choice of personas and the high correlation between LLM judgments and human annotations. Our work demonstrates that LLMs can serve as instruments to measure and improve scientific evaluation.

Figures (7)

• Figure 1: (1) LLM personas generate counterfactual writings in different rhetorical styles based on the same substantive content. (2) We calibrate the LLM personas' rhetorical scores via pairwise comparisons using an LLM judge. (3) We infer the rhetorical score of any query abstract by comparing it against the calibrated LLM persona panel.
• Figure 2: Left: Yearly trends in rhetorical scores. Points show yearly means with shaded bands indicating 95% confidence intervals. We notice a sharp increase since 2023. Right: We divide papers in the 2024-2025 batch into ten quantiles per rhetorical score and estimate their LLM usage via liang2024mapping. We notice a strong correlation between their rhetorical scores and the estimated LLM usage. More detailed statistics are presented in Table \ref{['tab:llm_usage_2024']} in Appendix.
• Figure 3: Persona win rates against the sampled 8,485 query papers.
• Figure 4: Mean rhetorical scores by subfield. Blue bars indicate subfield-specific mean values, with black horizontal error bars showing 95% confidence intervals based on the standard error of the mean. Subfield labels use standardized abbreviations of topic names, and numbers in parentheses indicate the number of papers in each subfield; see Appendix Table \ref{['table:name']} for the full topic mapping.
• Figure 5: Spearman Rank Correlation of Rhetorical Scores from Persona Subsets vs. Full 30-Persona Set. Results are averaged over 20 random trials for each subset size.

Theorems & Definitions (3)

• Definition 2.1: Stronger Rhetorical Style
• Definition 2.2: Weaker Rhetorical Style
• Definition 2.3: Order of Rhetorical Strength