Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

RADIUS: Ranking, Distribution, and Significance - A Comprehensive Alignment Suite for Survey Simulation

Weronika Łajewska, Paul Missault, George Davidson, Saab Mansour

Abstract

Simulation of surveys using LLMs is emerging as a powerful application for generating human-like responses at scale. Prior work evaluates survey simulation using metrics borrowed from other domains, which are often ad hoc, fragmented, and non-standardized, leading to results that are difficult to compare. Moreover, existing metrics focus mainly on accuracy or distributional measures, overlooking the critical dimension of ranking alignment. In practice, a simulation can achieve high accuracy while still failing to capture the option most preferred by humans - a distinction that is critical in decision-making applications. We introduce RADIUS, a comprehensive two-dimensional alignment suite for survey simulation that captures: 1) RAnking alignment and 2) DIstribUtion alignment, each complemented by statistical Significance testing. RADIUS highlights the limitations of existing metrics, enables more meaningful evaluation of survey simulation, and provides an open-source implementation for reproducible and comparable assessment.

RADIUS: Ranking, Distribution, and Significance - A Comprehensive Alignment Suite for Survey Simulation

Abstract

Simulation of surveys using LLMs is emerging as a powerful application for generating human-like responses at scale. Prior work evaluates survey simulation using metrics borrowed from other domains, which are often ad hoc, fragmented, and non-standardized, leading to results that are difficult to compare. Moreover, existing metrics focus mainly on accuracy or distributional measures, overlooking the critical dimension of ranking alignment. In practice, a simulation can achieve high accuracy while still failing to capture the option most preferred by humans - a distinction that is critical in decision-making applications. We introduce RADIUS, a comprehensive two-dimensional alignment suite for survey simulation that captures: 1) RAnking alignment and 2) DIstribUtion alignment, each complemented by statistical Significance testing. RADIUS highlights the limitations of existing metrics, enables more meaningful evaluation of survey simulation, and provides an open-source implementation for reproducible and comparable assessment.
Paper Structure (17 sections, 1 figure, 2 tables)

This paper contains 17 sections, 1 figure, 2 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Survey Simulation
  4. Alignment Suite
  5. Ranking Alignment
  6. Top Rank Match (TRM)
  7. Rank Correlation (RC)
  8. Distribution Alignment
  9. Total Variation Distance (TVD)
  10. Distribution Homogeneity (DH)
  11. Survey-Level Alignment
  12. Results
  13. Relative Strictness of Alignment Criteria
  14. Discriminative Power of Metrics
  15. Alignment Metrics Robustness
  16. ...and 2 more sections

Figures (1)

  • Figure 1: User simulation framework for surveys.