Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Too Open for Opinion? Embracing Open-Endedness in Large Language Models for Social Simulation

Bolei Ma, Yong Cao, Indira Sen, Anna-Carolina Haensch, Frauke Kreuter, Barbara Plank, Daniel Hershcovich

TL;DR

This position paper advocates embracing open-ended generation in LLM-based social simulations to better capture the diversity and reasoning of real populations. By connecting decades of social science open-ended survey practices with NLP capabilities, it argues that unconstrained text outputs enable richer measurement, exploration of minority views, and more realistic agent heterogeneity, while reducing researcher-imposed directive bias. The authors propose methodological frameworks that combine manual coding, qualitative analysis, and hybrid human–machine workflows, and outline concrete use-cases across populations, qualitative interviews, deliberation, and social-media simulations. They also discuss critical challenges—data, evaluation of infinite responses, and ethical risks—and call for benchmarks, multidimensional evaluation, bias-mitigation strategies, and transparent governance. Overall, the paper highlights practical avenues for integrating social science rigor with open-ended NLP to advance credible and versatile social simulations.

Abstract

Large Language Models (LLMs) are increasingly used to simulate public opinion and other social phenomena. Most current studies constrain these simulations to multiple-choice or short-answer formats for ease of scoring and comparison, but such closed designs overlook the inherently generative nature of LLMs. In this position paper, we argue that open-endedness, using free-form text that captures topics, viewpoints, and reasoning processes "in" LLMs, is essential for realistic social simulation. Drawing on decades of survey-methodology research and recent advances in NLP, we argue why this open-endedness is valuable in LLM social simulations, showing how it can improve measurement and design, support exploration of unanticipated views, and reduce researcher-imposed directive bias. It also captures expressiveness and individuality, aids in pretesting, and ultimately enhances methodological utility. We call for novel practices and evaluation frameworks that leverage rather than constrain the open-ended generative diversity of LLMs, creating synergies between NLP and social science.

Too Open for Opinion? Embracing Open-Endedness in Large Language Models for Social Simulation

TL;DR

This position paper advocates embracing open-ended generation in LLM-based social simulations to better capture the diversity and reasoning of real populations. By connecting decades of social science open-ended survey practices with NLP capabilities, it argues that unconstrained text outputs enable richer measurement, exploration of minority views, and more realistic agent heterogeneity, while reducing researcher-imposed directive bias. The authors propose methodological frameworks that combine manual coding, qualitative analysis, and hybrid human–machine workflows, and outline concrete use-cases across populations, qualitative interviews, deliberation, and social-media simulations. They also discuss critical challenges—data, evaluation of infinite responses, and ethical risks—and call for benchmarks, multidimensional evaluation, bias-mitigation strategies, and transparent governance. Overall, the paper highlights practical avenues for integrating social science rigor with open-ended NLP to advance credible and versatile social simulations.

Abstract

Large Language Models (LLMs) are increasingly used to simulate public opinion and other social phenomena. Most current studies constrain these simulations to multiple-choice or short-answer formats for ease of scoring and comparison, but such closed designs overlook the inherently generative nature of LLMs. In this position paper, we argue that open-endedness, using free-form text that captures topics, viewpoints, and reasoning processes "in" LLMs, is essential for realistic social simulation. Drawing on decades of survey-methodology research and recent advances in NLP, we argue why this open-endedness is valuable in LLM social simulations, showing how it can improve measurement and design, support exploration of unanticipated views, and reduce researcher-imposed directive bias. It also captures expressiveness and individuality, aids in pretesting, and ultimately enhances methodological utility. We call for novel practices and evaluation frameworks that leverage rather than constrain the open-ended generative diversity of LLMs, creating synergies between NLP and social science.
Paper Structure (16 sections, 2 figures, 1 table)

This paper contains 16 sections, 2 figures, 1 table.

Table of Contents

  1. Introduction
  2. Open-Endedness in LLM Social Simulation
  3. Current Practice and Limitations on Close-Ends in LLM Social Simulation
  4. Arguments for Open-Endedness
  5. Lessons from Human Studies
  6. Implications for LLM Social Simulations
  7. Use-Cases
  8. Practical Insights from Social Science and NLP
  9. Social Science $\rightarrow$ NLP
  10. NLP $\rightarrow$ Social Science
  11. Challenges and Future Directions
  12. Conclusion
  13. Limitations
  14. Ethical Considerations
  15. Detailed Use-Cases on Open-Ends in Social Simulations
  16. ...and 1 more sections

Figures (2)

  • Figure 1: Example of close-ended and open-ended surveying, illustrating that open-ended interaction better captures participant intentions, generates more diverse and explainable responses, and enables more accurate social simulation compared to a closed-ended one.
  • Figure 2: An example coding scheme modified for coding open-ended LLM responses to a question "In your opinion, what is the most important question facing Germany today?". The human-coders (or LLM-Coders) are instructed to categorize the open-ended responses into the listed classes.