Elicitron: An LLM Agent-Based Simulation Framework for Design Requirements Elicitation

TL;DR

Requirements elicitation is essential yet costly and prone to missing latent user needs. Elicitron leverages LLM agents to simulate diverse users, product experiences, and interviews, employing serial and parallel generation with diversity mechanisms to surface both explicit and latent needs. The study demonstrates that context-aware serial agent generation yields the most diverse coverage, and that LLM-based latent-need identification can outperform empathic lead-user interviews while remaining cost-effective. Together, these findings show that LLM agents can accelerate early-stage product development, reduce research costs, and drive design innovation by systematically uncovering nuanced user requirements.

Abstract

Requirements elicitation, a critical, yet time-consuming and challenging step in product development, often fails to capture the full spectrum of user needs. This may lead to products that fall short of expectations. This paper introduces a novel framework that leverages Large Language Models (LLMs) to automate and enhance the requirements elicitation process. LLMs are used to generate a vast array of simulated users (LLM agents), enabling the exploration of a much broader range of user needs and unforeseen use cases. These agents engage in product experience scenarios, through explaining their actions, observations, and challenges. Subsequent agent interviews and analysis uncover valuable user needs, including latent ones. We validate our framework with three experiments. First, we explore different methodologies for diverse agent generation, discussing their advantages and shortcomings. We measure the diversity of identified user needs and demonstrate that context-aware agent generation leads to greater diversity. Second, we show how our framework effectively mimics empathic lead user interviews, identifying a greater number of latent needs than conventional human interviews. Third, we showcase that LLMs can be used to analyze interviews, capture needs, and classify them as latent or not. Our work highlights the potential of using LLM agents to accelerate early-stage product development, reduce costs, and increase innovation.

Figures (5)

• Figure 1: Elictron's architecture for requirements elicitation using LLMs: First, LLM agents are generated within a design context in either serial and parallel fashion (incorporating diversity sampling to represent varied user perspectives). These agents then engage in simulated product experience scenarios, documenting each step (Action, Observation, Challenge) in detail. Following this, they undergo an agent interview process, where questions are asked and answered to surface latent user needs. In the final stage, latent needs are identified using an LLM on a provided criteria, and finally a report is generated from the identified latent needs.
• Figure 2: Four groups of users' embeddings after reducing dimensions to 2 using t-SNE. Group 1: Service and Conservation RGB]251,185,185(in red). Group 2: Outdoor Recreation and Camping RGB]188,203,229(in blue). Group 3: Adventure and Exploration RGB]185, 251, 185(in green). Group 4: Family Camping and Outdoor Activities RGB]220,200,231(in purple). The serial generation gives the best coverage of all four groups. Parallel generation with and without filtering both missed service and conservation-related users.
• Figure 3: The silhouette score measures the intra- and inter-cluster distance. The serial method results in stakeholder embeddings that are more difficult to cluster compared to the parallel and parallel with filtering methods, which indicates that the serial embeddings are more diverse.
• Figure 4: Comparison of the average number of latent needs identified by each user agent across the experimental conditions. The error bars indicate standard deviation with n=20 for each condition.
• Figure 5: Comparative confusion matrices for latent need identification: (a) zero-Shot classification, (b) classification with latent need criteria, and (c) classification employing a chain-of-thought approach and latent need criteria.