Generative Artificial Intelligence-Guided User Studies: An Application for Air Taxi Services

TL;DR

This study tackles the challenge of safe, rapid, and iterative UX evaluation for emerging transportation tech by leveraging Generative AI. A GPT-4–driven design-thinking pipeline produces Air Taxi Journey (ATJ) scenarios, visuals via Midjourney/Runway, and a structured questionnaire evaluated by 72 real participants. Results show that ATJ exposure improves willingness and satisfaction, with education and gender influencing effects, and demonstrate the potential of LLMs to simulate participant responses as a pretest tool. The approach reduces safety concerns and resource demands in early design phases and can generalize to other high-risk domains, offering a scalable path for rapid, AI-assisted UX development.

Abstract

User studies are crucial for meeting user needs. In user studies, real experimental scenarios and participants are constructed and recruited. However, emerging and unfamiliar studies face limitations, including safety concerns and iterative efficiency. To address these challenges, this study utilises a Generative Artificial Intelligence (GenAI) to create GenAI-generated scenarios for user experience (UX). By recruiting real users to evaluate this experience, we can collect feedback that enables rapid iteration in the early design phase. The air taxi is particularly representative of these challenges and has been chosen as the case study for this research. The key contribution was designing an Air Taxi Journey (ATJ) using Large Language Models (LLMs) and AI image and video generators. Based on the GPT-4-generated scripts, key visuals were created for the air taxi, and the ATJ was evaluated by 72 participants. Furthermore, the LLMs demonstrated the ability to identify and suggest environments that significantly improve participants' willingness toward air taxis. Education level and gender significantly influenced participants' the difference in willingness and their satisfaction with the ATJ. Satisfaction with the ATJ serves as a mediator, significantly influencing participants' willingness to take air taxis. Our study confirms the capability of GenAI to support user studies, providing a feasible approach and valuable insights for designing air taxi UX in the early design phase.

Figures (16)

• Figure 1: The summary of this study. (a) The main contribution of this study. [I] means a user study conducted by real experimental scenarios and real users, [II] means a user study conducted by virtual experimental scenarios and real users, [III] means a user study conducted by real experimental scenarios and virtual users. [IV] means a user study conducted by virtual experimental scenarios and virtual users. This study focus on a user study with virtual experimental scenarios (air taxi) and real users. (b) The research structure of this study.
• Figure 2: Procedure of how we conducted the user study using LLM and AI image and video generators. A, B, C, D, and E represent the five-step iterative design thinking process respectively, including empathize, define, ideate, prototype, and test. A1, B1, C1, and D1 represent the prompts we designed for each step. A2, B2, C2, and D2 represent the outputs from GPT-4. A3, B3, C3, and D3 represent the inputs into GPT-4. In the E-Test step, E1 is the necessary test content material for the user test. E2 is the real responses from participants’ evaluation towards ATJ. E3 represents the data analysis of towards evaluated responses step.
• Figure 3: A customer journey map. It is mainly summarized based on the selected mockup (in Appendix C - Table \ref{['tab:freq-5']}) and storyboard (in Appendix C - Table \ref{['tab:freq-6']})
• Figure 4: Results of a quantitative analysis of influencing factors affecting participants’ attitude change with ATJ. The four subfigures therein represent the different subgroups of participants in terms of gender, education level, age, and employed status. The vertical axis represents the mean of participants’ willingness to take an air taxi. *$p\textless.05$; **$p\textless.01$; ***$p\textless.001$
• Figure 5: Results of a quantitative analysis of influencing factors affecting participants’ satisfaction with ATJ. The vertical axis represents the participants’ satisfaction ratings of ATJ. The horizontal axis shows the different subgroups of the participants in terms of gender, education level, age, and employment status. *$p\textless.05$; **$p\textless.01$; ***$p\textless.001$