Feed-O-Meter: Investigating AI-Generated Mentee Personas as Interactive Agents for Scaffolding Design Feedback Practice

TL;DR

Feed-O-Meter introduces a large-language-model powered environment for practicing design feedback via role-playing as a mentor to an AI mentee. It uses a feedback reflection interface to visualize the impact of feedback on the mentee's idea development and to prompt deeper reflection. In a within-subject study (N=24), users reported higher engagement, improved feedback specificity/justification/action, and greater perceived efficacy when using Feed-O-Meter compared to baseline. The work discusses design considerations for integrating AI personas in design education and outlines future directions for long-term deployment and multimodal feedback practice.

Abstract

Effective feedback, including critique and evaluation, helps designers develop design concepts and refine their ideas, supporting informed decision-making throughout the iterative design process. However, in studio-based design courses, students often struggle to provide feedback due to a lack of confidence and fear of being judged, which limits their ability to develop essential feedback-giving skills. Recent advances in large language models (LLMs) suggest that role-playing with AI agents can let learners engage in multi-turn feedback without the anxiety of external judgment or the time constraints of real-world settings. Yet prior studies have raised concerns that LLMs struggle to behave like real people in role-play scenarios, diminishing the educational benefits of these interactions. Therefore, designing AI-based agents that effectively support learners in practicing and developing intellectual reasoning skills requires more than merely assigning the target persona's personality and role to the agent. By addressing these issues, we present Feed-O-Meter, a novel system that employs carefully designed LLM-based agents to create an environment in which students can practice giving design feedback. The system enables users to role-play as mentors, providing feedback to an AI mentee and allowing them to reflect on how that feedback impacts the AI mentee's idea development process. A user study (N=24) indicated that Feed-O-Meter increased participants' engagement and motivation through role-switching and helped them adjust feedback to be more comprehensible for an AI mentee. Based on these findings, we discuss future directions for designing systems to foster feedback skills in design education.

Figures (8)

• Figure 1: The main user interface of the Feed-O-Meter. (A) The Idea Proposal Interface displays predefined design topics, goals, and the AI mentee's current design idea. By clicking the "Update Idea" button (A1), the user can request the AI mentee to update its design idea based on feedback. (B) Chat Interface allows users to provide feedback to the AI mentee. (C) Feedback Reflection Interface includes the Mentee's Profile (C1), showing the mentee's progress, and the Feedback Evaluation Dashboard (C2), which visualizes feedback criteria. The timer (D) tracks the session duration.
• Figure 2: Structure of the Feed-O-Meter's baseline pipeline. (1) Feedback is provided by the user and processed by the response generator through the following steps. (2) The categorizer categorizes the feedback into six predefined categories, such as information and recommendations. (3) knowledge and action plan are extracted by the knowledge extractor according to their categories and integrated into the knowledge state. When the user clicks the "Update Idea" button, a design idea is revised based on action plans and the chat history.
• Figure 3: Pipeline of the Feedback Reflection Interface. The pipeline starts by categorizing user feedback into one of six categories—three from the question family and three from the statement family. Each feedback sentence is then evaluated according to criteria specific to its category. The evaluation results are displayed in the feedback reflection interface, influencing the mentee's facial expressions, which change dynamically based on the feedback. Counter-questions are generated when certain conditions are met.
• Figure 4: The outline of the user study with the time allocated for each step.
• Figure 5: Comparison of expert evaluation of participants' feedback under the Feed-O-Meter condition and the baseline condition. For sentence-level, question-based feedback was evaluated by timeliness, goal relevance, and level, while statement-based feedback was evaluated by specificity, justification, and action. The overall feedback session was evaluated by ratio of divergent and convergent, ratio of question and statement, and overall helpfulness. (- : $p > .05$, $\ast$ : $p < .050$, $\ast\ast$ : $p < .010$, $\ast\ast \ast$ : $p < .001$)