Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

The Potential and Challenges of Evaluating Attitudes, Opinions, and Values in Large Language Models

Bolei Ma, Xinpeng Wang, Tiancheng Hu, Anna-Carolina Haensch, Michael A. Hedderich, Barbara Plank, Frauke Kreuter

TL;DR

This paper aims to bridge the gap by providing a comprehensive overview of recent works on the evaluation of AOVs in LLMs, and addresses the potential and challenges with respect to understanding the model, human-AI alignment, and downstream application in social sciences.

Abstract

Recent advances in Large Language Models (LLMs) have sparked wide interest in validating and comprehending the human-like cognitive-behavioral traits LLMs may capture and convey. These cognitive-behavioral traits include typically Attitudes, Opinions, Values (AOVs). However, measuring AOVs embedded within LLMs remains opaque, and different evaluation methods may yield different results. This has led to a lack of clarity on how different studies are related to each other and how they can be interpreted. This paper aims to bridge this gap by providing a comprehensive overview of recent works on the evaluation of AOVs in LLMs. Moreover, we survey related approaches in different stages of the evaluation pipeline in these works. By doing so, we address the potential and challenges with respect to understanding the model, human-AI alignment, and downstream application in social sciences. Finally, we provide practical insights into evaluation methods, model enhancement, and interdisciplinary collaboration, thereby contributing to the evolving landscape of evaluating AOVs in LLMs.

The Potential and Challenges of Evaluating Attitudes, Opinions, and Values in Large Language Models

TL;DR

This paper aims to bridge the gap by providing a comprehensive overview of recent works on the evaluation of AOVs in LLMs, and addresses the potential and challenges with respect to understanding the model, human-AI alignment, and downstream application in social sciences.

Abstract

Recent advances in Large Language Models (LLMs) have sparked wide interest in validating and comprehending the human-like cognitive-behavioral traits LLMs may capture and convey. These cognitive-behavioral traits include typically Attitudes, Opinions, Values (AOVs). However, measuring AOVs embedded within LLMs remains opaque, and different evaluation methods may yield different results. This has led to a lack of clarity on how different studies are related to each other and how they can be interpreted. This paper aims to bridge this gap by providing a comprehensive overview of recent works on the evaluation of AOVs in LLMs. Moreover, we survey related approaches in different stages of the evaluation pipeline in these works. By doing so, we address the potential and challenges with respect to understanding the model, human-AI alignment, and downstream application in social sciences. Finally, we provide practical insights into evaluation methods, model enhancement, and interdisciplinary collaboration, thereby contributing to the evolving landscape of evaluating AOVs in LLMs.
Paper Structure (57 sections, 7 figures, 2 tables)

This paper contains 57 sections, 7 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Related Survey Papers.
  3. Our Survey Paper.
  4. Definitions
  5. An Overview of Related Works for AOVs in LLMs
  6. Attitudes/Opinions
  7. US-Centric Public Opinion Polls.
  8. Non-US-Centric Public Opinion Polls.
  9. Additional Data Sources Used.
  10. Values
  11. Value Orientation of LLMs.
  12. Curated Datasets and Frameworks.
  13. Other Related Topics
  14. How LLMs Are Queried for AOVs
  15. Input
  16. ...and 42 more sections

Figures (7)

  • Figure 1: A taxonomy of evaluation pipeline across input inbox $\rightarrow$ model robot $\rightarrow$ output comment $\rightarrow$ evaluation chart-bar.
  • Figure 2: A simple model of the survey response process groves2004survey
  • Figure 3: Distribution of the deployed models in the surveyed works.
  • Figure 4: An example of a simple close-ended question with a general system instruction prompt wang2024my.
  • Figure 5: An example of a close-ended question with a predefined persona and several opinions together as input prompt hwang-etal-2023-aligning.
  • ...and 2 more figures