Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Characterizing the ability of LLMs to recapitulate Americans' distributional responses to public opinion polling questions across political issues

Eric Gong, Nathan E. Sanders, Bruce Schneier

Abstract

Traditional survey-based political issue polling is becoming less tractable due to increasing costs and risk of bias associated with growing non-response rates and declining coverage of key demographic groups. With researchers and pollsters seeking alternatives, Large Language Models have drawn attention for their potential to augment human population studies in polling contexts. We propose and implement a new framework for anticipating human responses on multiple-choice political issue polling questions by directly prompting an LLM to predict a distribution of responses. By comparison to a large and high quality issue poll of the US population, the Cooperative Election Study, we evaluate how the accuracy of this framework varies across a range of demographics and questions on a variety of topics, as well as how this framework compares to previously proposed frameworks where LLMs are repeatedly queried to simulate individual respondents. We find the proposed framework consistently exhibits more accurate predictions than individual querying at significantly lower cost. In addition, we find the performance of the proposed framework varies much more systematically and predictably across demographics and questions, making it possible for those performing AI polling to better anticipate model performance using only information available before a query is issued.

Characterizing the ability of LLMs to recapitulate Americans' distributional responses to public opinion polling questions across political issues

Abstract

Traditional survey-based political issue polling is becoming less tractable due to increasing costs and risk of bias associated with growing non-response rates and declining coverage of key demographic groups. With researchers and pollsters seeking alternatives, Large Language Models have drawn attention for their potential to augment human population studies in polling contexts. We propose and implement a new framework for anticipating human responses on multiple-choice political issue polling questions by directly prompting an LLM to predict a distribution of responses. By comparison to a large and high quality issue poll of the US population, the Cooperative Election Study, we evaluate how the accuracy of this framework varies across a range of demographics and questions on a variety of topics, as well as how this framework compares to previously proposed frameworks where LLMs are repeatedly queried to simulate individual respondents. We find the proposed framework consistently exhibits more accurate predictions than individual querying at significantly lower cost. In addition, we find the performance of the proposed framework varies much more systematically and predictably across demographics and questions, making it possible for those performing AI polling to better anticipate model performance using only information available before a query is issued.
Paper Structure (26 sections, 6 figures, 6 tables)

This paper contains 26 sections, 6 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Methodology
  3. Target question and demographic selection
  4. Opinion prediction with large language models
  5. Quantitatively differentiating questions by topic
  6. Processing human responses
  7. Comparing human and LLM predicted responses
  8. Results
  9. Single Individual and Direct Distribution querying frameworks
  10. Direct Distribution framework performance
  11. Demographic and query related coefficients
  12. Vector embedding feature coefficients
  13. Predicting performance per question
  14. Discussions
  15. Conclusion
  16. ...and 11 more sections

Figures (6)

  • Figure 1: Framework for curating and evaluating LLM predictions on opinion polling questions
  • Figure 2: Vector embedding of survey questions mapped to two dimensional space with UMAP and color coded by LLM-assigned content tag.
  • Figure 3: Histograms of pair-wise metric differences between SI and DD Frameworks
  • Figure 4: LLM response heterogeneity (standard deviation) for the DD and SI frameworks versus the heterogeneity of human responses. Moving average and associated 2$\sigma$ Standard Error shown
  • Figure 5: DD and SI framework SDD metric as a function of NEMD, with instances in which the LLM returned the same response across all trials (Standard Deviation of 0) for SI framework identified.
  • ...and 1 more figures