Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Crowdsourced Adaptive Surveys

Yamil Velez

TL;DR

The paper addresses the slow adaptation of traditional surveys to evolving information environments by introducing the crowdsourced adaptive survey (CSAS) method, which uses open-ended responses to generate a dynamic bank of survey items. It combines GPT-based item generation with a Gaussian Thompson Sampling multi-armed bandit to iteratively prioritize questions, allowing real-time exploration of emerging topics. The method is demonstrated across three domains—national issue importance, misinformation within Latino communities, and local political concerns—showing CSAS can reveal topics that standard instruments may miss and adapt to subpopulation nuances. The work also discusses integration with traditional survey design, operational costs, and broader implications for democratizing instrument development and research participation.

Abstract

Public opinion surveys are vital for informing democratic decision-making, but responding to rapidly evolving information environments and measuring beliefs within niche communities can be challenging for traditional survey methods. This paper introduces a crowdsourced adaptive survey methodology (CSAS) that unites advances in natural language processing and adaptive algorithms to generate question banks that evolve with user input. The CSAS method converts open-ended text provided by participants into survey items and applies a multi-armed bandit algorithm to determine which questions should be prioritized in the survey. The method's adaptive nature allows for the exploration of new survey questions, while imposing minimal costs in survey length. Applications in the domains of Latino information environments, national issue importance, and local politics showcase CSAS's ability to identify topics that might otherwise escape the notice of survey researchers. I conclude by highlighting CSAS's potential to bridge conceptual gaps between researchers and participants in survey research.

Crowdsourced Adaptive Surveys

TL;DR

The paper addresses the slow adaptation of traditional surveys to evolving information environments by introducing the crowdsourced adaptive survey (CSAS) method, which uses open-ended responses to generate a dynamic bank of survey items. It combines GPT-based item generation with a Gaussian Thompson Sampling multi-armed bandit to iteratively prioritize questions, allowing real-time exploration of emerging topics. The method is demonstrated across three domains—national issue importance, misinformation within Latino communities, and local political concerns—showing CSAS can reveal topics that standard instruments may miss and adapt to subpopulation nuances. The work also discusses integration with traditional survey design, operational costs, and broader implications for democratizing instrument development and research participation.

Abstract

Public opinion surveys are vital for informing democratic decision-making, but responding to rapidly evolving information environments and measuring beliefs within niche communities can be challenging for traditional survey methods. This paper introduces a crowdsourced adaptive survey methodology (CSAS) that unites advances in natural language processing and adaptive algorithms to generate question banks that evolve with user input. The CSAS method converts open-ended text provided by participants into survey items and applies a multi-armed bandit algorithm to determine which questions should be prioritized in the survey. The method's adaptive nature allows for the exploration of new survey questions, while imposing minimal costs in survey length. Applications in the domains of Latino information environments, national issue importance, and local politics showcase CSAS's ability to identify topics that might otherwise escape the notice of survey researchers. I conclude by highlighting CSAS's potential to bridge conceptual gaps between researchers and participants in survey research.
Paper Structure (14 sections, 3 figures, 1 table)

This paper contains 14 sections, 3 figures, 1 table.

Table of Contents

  1. Introduction
  2. Dynamic Survey Methodologies: Existing Approaches
  3. Computerized Adaptive Tests
  4. Wiki Surveys
  5. The Crowdsourced Adaptive Survey Method
  6. An Application to Issue Importance
  7. Results
  8. An Application to Latino Information Environments
  9. Results
  10. An Application to Local Political Issues
  11. Concerns and Caveats
  12. Is the CSAS method compatible with traditional survey design?
  13. Late Arrivals, Prompting Participants, and Costs of LLM Inference
  14. Conclusion

Figures (3)

  • Figure 1: CSAS Process Flowchart and Representative Set of Issues uncovered by CSAS.
  • Figure 2: IPW-weighted estimates of survey questions measuring issue importance with corresponding 95% confidence intervals.
  • Figure 3: IPW-weighted estimates of survey questions measuring negative political claims with corresponding 95% confidence intervals. Items in gray were initial seed items based on fact-checked claims produced by Latino-focused fact-checking organizations. Black items are participant-generated items.