From Answer Givers to Design Mentors: Guiding LLMs with the Cognitive Apprenticeship Model

TL;DR

The paper tackles turning AI chatbots into design mentors to promote reflective design reasoning in visualization tasks. It introduces DesignMentor, a GPT-based system guided by the Cognitive Apprenticeship Model (CAM) and seven design guidelines that operationalize CAM’s six methods. Through a within-subjects study with 24 visualization practitioners, DesignMentor markedly increases design reasoning, the depth and breadth of feedback, and metacognitive engagement, though it also imposes higher cognitive effort and yields preferences that depend on the design phase. The work contributes a design mentorship codebook, practical prompt-instruction guidelines, and empirical evidence for when guided AI mentorship improves outcomes versus when direct answers are preferred, with implications for adaptive AI support in education and practice. Limitations include the need for multimodal feedback and longitudinal studies, as well as generalizability to other design domains.

Abstract

Design feedback helps practitioners improve their artifacts while also fostering reflection and design reasoning. Large Language Models (LLMs) such as ChatGPT can support design work, but often provide generic, one-off suggestions that limit reflective engagement. We investigate how to guide LLMs to act as design mentors by applying the Cognitive Apprenticeship Model, which emphasizes demonstrating reasoning through six methods: modeling, coaching, scaffolding, articulation, reflection, and exploration. We operationalize these instructional methods through structured prompting and evaluate them in a within-subjects study with data visualization practitioners. Participants interacted with both a baseline LLM and an instructional LLM designed with cognitive apprenticeship prompts. Surveys, interviews, and conversational log analyses compared experiences across conditions. Our findings show that cognitively informed prompts elicit deeper design reasoning and more reflective feedback exchanges, though the baseline is sometimes preferred depending on task types or experience levels. We distill design considerations for AI-assisted feedback systems that foster reflective practice.

Figures (5)

• Figure 1: A diagram of a three-stage feedback process showing the roles of Mentors and Mentees. In Phase 1, the mentee articulates their design problem through Articulating and Bounding. In Phase 2, the mentor diagnoses and provides support through Scoping, Coaching, Modeling, and Scaffolding. In Phase 3, the mentee explores solutions and reflects on the process through Exploring and Reflecting.
• Figure 2: Case study. A usage scenario of feedback session with DesignMentor discussing color choice and design improvement over energy visualization. a) DesignMentor initiates the feedback session through a structured introduction and process overview. b-c) Moving on to Phase 2, it guides users to articulate the design goal and context, and specify and confirm their inquiries into the form of questions. d-f) Throughout discussing the question 1 about the color choice, DesignMentor provides gradual supports from hints to demonstration, with its three mentorship practices including support, affirm, and confirm. In Phase 3, it prompts users to reflect on the feedback session by comparing design ideas and decisions with the existing visualizations, which helps envision the next steps in the design process.
• Figure 3: The overview of evaluation questions and analysis methods. The design of evaluation questions build on the core inquiry of our study: "Does design reasoning facilitate better feedback process and outcome?" The three evaluation questions altogether explore DesignMentor's effectiveness on promoting design reasoning and feedback outcome to quantitatively and qualitatively scrutinize what leads to their preferences over either system.
• Figure 4: A response matrix heatmap showing the pre-survey results from 24 participants on the capabilities of existing AI chatbots. This highlights that, when discussing design problems and decisions with existing AI chatbots, they often lack in providing self-regulation, indicating AI chatbots do not encourage articulation and self-reflection on design choices, while relatively giving a clear guidance on the complex visualization tasks (i.e., problem guidance) and clearly explain potential solutions (i.e., transparency).
• Figure 5: A series of dot plots comparing post-task survey ratings for DesignMentor (orange) and the ChatGPT-4o baseline (blue) from 25 participants. The chart shows mean scores on a 5-point scale for 16 items, which are grouped into four categories: Feedback Completeness, Feedback Level, User Experience, and Delivery Quality. Error bars represent 95% confidence intervals. The main finding is that for nearly all positive metrics, such as "Feedup", "Feedback", and "Self Regulate", DesignMentor's scores are visibly higher than the baseline's.