Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Methodological Foundations for AI-Driven Survey Question Generation

Ted K. Mburu, Kangxuan Rong, Campbell J. McColley, Alexandra Werth

TL;DR

The paper presents a methodological framework for deploying AI-driven survey instruments in engineering education, centering on the Synthetic Question-Response Analysis (SQRA) framework and Activity Theory to pre-validate questions before human deployment. It demonstrates integration of OpenAI GPT models within Qualtrics to generate adaptive, context-aware prompts and uses AI-to-AI simulations with simulated personas to iteratively refine prompts and reduce biases. Through sentiment, cosine similarity, and structural analyses, it contrasts AI-to-AI and AI-to-Human interactions, revealing contextually relevant question generation but also issues such as redundancy, double-barreled questions, and jargon, which prompt careful prompt engineering. The work highlights the promise of AI-driven surveys for scalable, personalized data collection while underscoring ethical considerations, data privacy, and the need for ongoing validation, model updates, and potential hybrid approaches combining AI with follow-up human inquiry.

Abstract

This paper presents a methodological framework for using generative AI in educational survey research. We explore how Large Language Models (LLMs) can generate adaptive, context-aware survey questions and introduce the Synthetic Question-Response Analysis (SQRA) framework, which enables iterative testing and refinement of AI-generated prompts prior to deployment with human participants. Guided by Activity Theory, we analyze how AI tools mediate participant engagement and learning, and we examine ethical issues such as bias, privacy, and transparency. Through sentiment, lexical, and structural analyses of both AI-to-AI and AI-to-human survey interactions, we evaluate the alignment and effectiveness of these questions. Our findings highlight the promise and limitations of AI-driven survey instruments, emphasizing the need for robust prompt engineering and validation to support trustworthy, scalable, and contextually relevant data collection in engineering education.

Methodological Foundations for AI-Driven Survey Question Generation

TL;DR

The paper presents a methodological framework for deploying AI-driven survey instruments in engineering education, centering on the Synthetic Question-Response Analysis (SQRA) framework and Activity Theory to pre-validate questions before human deployment. It demonstrates integration of OpenAI GPT models within Qualtrics to generate adaptive, context-aware prompts and uses AI-to-AI simulations with simulated personas to iteratively refine prompts and reduce biases. Through sentiment, cosine similarity, and structural analyses, it contrasts AI-to-AI and AI-to-Human interactions, revealing contextually relevant question generation but also issues such as redundancy, double-barreled questions, and jargon, which prompt careful prompt engineering. The work highlights the promise of AI-driven surveys for scalable, personalized data collection while underscoring ethical considerations, data privacy, and the need for ongoing validation, model updates, and potential hybrid approaches combining AI with follow-up human inquiry.

Abstract

This paper presents a methodological framework for using generative AI in educational survey research. We explore how Large Language Models (LLMs) can generate adaptive, context-aware survey questions and introduce the Synthetic Question-Response Analysis (SQRA) framework, which enables iterative testing and refinement of AI-generated prompts prior to deployment with human participants. Guided by Activity Theory, we analyze how AI tools mediate participant engagement and learning, and we examine ethical issues such as bias, privacy, and transparency. Through sentiment, lexical, and structural analyses of both AI-to-AI and AI-to-human survey interactions, we evaluate the alignment and effectiveness of these questions. Our findings highlight the promise and limitations of AI-driven survey instruments, emphasizing the need for robust prompt engineering and validation to support trustworthy, scalable, and contextually relevant data collection in engineering education.
Paper Structure (36 sections, 18 figures, 7 tables)

This paper contains 36 sections, 18 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Background
  3. Emergence of Generative AI in Engineering Education Research
  4. Limitations and Ethical Concerns of AI-Driven Question Generation
  5. Activity Theory as a Theoretical Framework
  6. AT for AI-Driven Survey Design
  7. AT for AI-Driven Survey Validation
  8. Methodology
  9. Survey Design and AI Integration
  10. The Synthetic Question-Response Analysis (SQRA) Framework
  11. Prompt Engineering
  12. Question Generation Prompt
  13. Participant Persona Prompts
  14. Comparing SQRA Framework to in situ Application of AI-Generated Questions
  15. Sentiment Analysis
  16. ...and 21 more sections

Figures (18)

  • Figure 1: Comparison of traditional surveys, interviews, and AI-driven surveys highlighting their benefits, challenges, and opportunities.
  • Figure 2: Adaptation of Engeström's AT triangle for AI-driven surveys.
  • Figure 3: System prompt structure for AI-generated questions and personas.
  • Figure 4: Synthetic Question-Response Analysis (SQRA) framework.
  • Figure 5: Workflow diagram for integrating LLM into Qualtrics.
  • ...and 13 more figures