Large Language Models Can Be Used to Estimate the Latent Positions of Politicians

TL;DR

The liberal-conservative scale, used to validate LLM-driven scaling, strongly correlates with existing measures and offsets interpretive gaps, suggesting LLMs synthesize relevant data from internet and digitized media rather than memorizing existing measures.

Abstract

Existing approaches to estimating politicians' latent positions along specific dimensions often fail when relevant data is limited. We leverage the embedded knowledge in generative large language models (LLMs) to address this challenge and measure lawmakers' positions along specific political or policy dimensions. We prompt an instruction/dialogue-tuned LLM to pairwise compare lawmakers and then scale the resulting graph using the Bradley-Terry model. We estimate novel measures of U.S. senators' positions on liberal-conservative ideology, gun control, and abortion. Our liberal-conservative scale, used to validate LLM-driven scaling, strongly correlates with existing measures and offsets interpretive gaps, suggesting LLMs synthesize relevant data from internet and digitized media rather than memorizing existing measures. Our gun control and abortion measures -- the first of their kind -- differ from the liberal-conservative scale in face-valid ways and predict interest group ratings and legislator votes better than ideology alone. Our findings suggest LLMs hold promise for solving complex social science measurement problems.

Figures (8)

• Figure 1: An overview of the proposed pairwise comparison approach with instruction/dialogue-tuned generative LLMs.
• Figure 2: First Dimension of DW-NOMINATE vs. Ideology LaMP scores. Democratic senators are in blue, Republican senators are in red, and Independent senators are in green.
• Figure 3: Correlation matrices of LaMP scores, the first dimension of DW-NOMINATE, hopkins_noel_2022's perceived ideology scores, and bonica2013's CFscores.
• Figure 4: Comparing the proportion of variance, $R^2$, between the full and reduced models. The full model regresses perceived ideology scores on both the first dimension of DW-NOMINATE and Ideology LaMP scores. The reduced models only use DW-NOMINATE (denoted as "Only DW-NOMINATE") or Ideology LaMP scores (denoted as "Only Ideology LaMP scores") as predictors. The proportion of variance explained, $R^2$, is always lower when Ideology LaMP scores are removed as a predictor than when DW-NOMINATE is removed.
• Figure 5: Gun Control LaMP scores across all senators. Democratic senators are in blue, Republican senators are in red, and Independent senators are in green.