A Design-based Solution for Causal Inference with Text: Can a Language Model Be Too Large?

TL;DR

This paper addresses the challenge of causal inference for linguistic properties, focusing on latent confounding and overlap when text features encode the treatment $T$. It introduces a design-based experimental approach that uses study participants to generate texts and editors to flip $T$ while fixing other content, enabling unbiased estimation of $\tau_t$ despite latent confounding. Through simulations and a large real-data experiment, it shows that language-model–based estimators (TextCause, TI) can cause positivity violations and bias, while simpler bag-of-words approaches often perform more robustly; the authors provide a practical, auditable benchmark for evaluating text-as-treatment estimators. Substantively, expressing intellectual humility in political arguments reduces perceived aggressiveness but also lowers informativeness and persuasiveness, revealing a nuanced trade-off for using humility to combat polarization. The work offers methodological advances with broad implications for social platforms, policymakers, and causal-text research, by delivering an auditable design and ground-truth validation framework that can guide future methodological developments.

Abstract

Many social science questions ask how linguistic properties causally affect an audience's attitudes and behaviors. Because text properties are often interlinked (e.g., angry reviews use profane language), we must control for possible latent confounding to isolate causal effects. Recent literature proposes adapting large language models (LLMs) to learn latent representations of text that successfully predict both treatment and the outcome. However, because the treatment is a component of the text, these deep learning methods risk learning representations that actually encode the treatment itself, inducing overlap bias. Rather than depending on post-hoc adjustments, we introduce a new experimental design that handles latent confounding, avoids the overlap issue, and unbiasedly estimates treatment effects. We apply this design in an experiment evaluating the persuasiveness of expressing humility in political communication. Methodologically, we demonstrate that LLM-based methods perform worse than even simple bag-of-words models using our real text and outcomes from our experiment. Substantively, we isolate the causal effect of expressing humility on the perceived persuasiveness of political statements, offering new insights on communication effects for social media platforms, policy makers, and social scientists.

Figures (7)

• Figure 1: The top (bottom) panel shows estimates computed on the filtered datasets with the original (confounded) outcomes. We estimated the true average treatment effect (ATE) as described in Proposition 1. The 2.5 to 97.5 percentiles of these estimates across 100 replicas are shown as gray bands in each subplot. Each subplot features boxplots of the ATE estimated by each method for the 100 data replicas. The facet columns indicate the outcome label investigated in each experiment.
• Figure 2: Causal diagram of document labels for the latent treatment $D$, words $W$, the latent treatment itself $T$, other latent content $Z$, and outcome $Y$. Throughout this section, we assume that $D$ and $T$ are binary. Shaded nodes are observed, non-shaded nodes are unobserved but (potentially) measurable from $W$. There are three relevant potential outcomes that are equivalent under the standard SUTVA assumption rubin1980randomization. First, $Y_i(D)$ is unit $i$'s outcome depending on assigned document label $D$. Second, $Y_i(W) = Y_i(W(D))$, the outcome depending on the words in the document read by unit $i$, which are determined by $D$. Third, $Y_i(T,Z) = Y_i(T(W),Z(W))$, the outcome dependent on the latent features in the words, which are of course determined by the words themselves.
• Figure 3: X axis displays whether a text was an original or an edit. The y-axis represents the IH rating of each text on a scale from 0-100. Each line links the IH rating for an (original, edit) pair. Texts that are originally IH should see a decreasing line since the edit should not be IH. In contrast, texts that are originally not IH should see an increasing line since the edit should be IH. Lines that follow the expected slope are blue, dotted, and faded while those that are wrong are in red, solid, and bold.
• Figure 4: 95% confidence intervals (y-axis) showing the average treatment effect of intellectual humility on each outcome (x-axis). The dotted line represents no effect at 0.
• Figure 5: Intellectual humility definition in survey.

Theorems & Definitions (1)

• Proposition 4.1