The Metacognitive Demands and Opportunities of Generative AI

TL;DR

This paper argues that the usability challenges of generative AI arise from metacognitive demands placed on users. It adopts a metacognition framework that separates knowledge/experiences from monitoring/control, and applies it to prompting, evaluating, and automation decisions in GenAI use. The authors propose two design directions: (i) implement metacognitive support strategies (planning, self-evaluation, self-management) to enhance users’ metacognition, and (ii) reduce metacognitive load via improved explainability and system customizability, while managing overall cognitive load. They review evidence from cognitive science and GenAI user studies, outline measurement approaches for metacognition, and present open research questions and hypothetical interventions. The work highlights how GenAI’s model flexibility, generality, and originality can be leveraged to improve human-AI interaction by actively supporting metacognition and designing for reduced cognitive burden, with implications for HCI, AI design, and applied cognition research.

Abstract

Generative AI (GenAI) systems offer unprecedented opportunities for transforming professional and personal work, yet present challenges around prompting, evaluating and relying on outputs, and optimizing workflows. We argue that metacognition$\unicode{x2013}$the psychological ability to monitor and control one's thoughts and behavior$\unicode{x2013}$offers a valuable lens to understand and design for these usability challenges. Drawing on research in psychology and cognitive science, and recent GenAI user studies, we illustrate how GenAI systems impose metacognitive demands on users, requiring a high degree of metacognitive monitoring and control. We propose these demands could be addressed by integrating metacognitive support strategies into GenAI systems, and by designing GenAI systems to reduce their metacognitive demand by targeting explainability and customizability. Metacognition offers a coherent framework for understanding the usability challenges posed by GenAI, and provides novel research and design directions to advance human-AI interaction.

Figures (5)

• Figure 1: A simplified descriptive framework for metacognition. Metacognitive knowledge is the explicit understanding of one's abilities, strategies, and beliefs. Metacognitive experiences include things that people can directly experience, such as a feeling of familiarity or other implicit cues that provide information about cognitive processes. Metacognitive knowledge and experiences are interrelated in that experiences can become encoded as knowledge, and knowledge can be retrieved during experiences (§\ref{['subsec:MCinter']}). Both of these can influence (and be influenced by) metacognitive monitoring, which includes self-awareness, and confidence and its adjustment. Metacognitive monitoring, in turn, influences (and is influenced by) metacognitive control processes, such as metacognitive flexibility and task decomposition. Metacognitive control acts upon the (object-level) cognitive processes involved in a task. Arrows indicate directions of influence (§\ref{['subsec:MCinter']}).
• Figure 2: Metacognitive demands posed by generative AI at each point in a simplified user workflow. Often embedded within a workflow with manual tasks, users may first need to formulate a prompt, requiring metacognitive abilities including self-awareness of task goals and task decomposition (a). Systems that provide automated suggestions such as GitHub Copilot alleviate some of the demands associated with prompting. Depending on the output, iterating on the prompt may be necessary, which requires well-adjusted confidence in one's prompting ability and metacognitive flexiblity to adapt prompting strategies as necessary (b). Likewise, evaluating the output requires well-adjusted confidence in one's ability to judge its validity (c). Beyond the local interaction with a GenAI system, there is an overarching demand connected to understanding whether, when, and how to apply GenAI to one's workflows---one's 'automation strategy'. This requires self-awareness of how GenAI applies to and affects one's workflows, and well-adjusted confidence in the ability to complete tasks manually and with GenAI (d). Finally, it also requires metacognitive flexbility to adapt one's workflows as necessary (e).
• Figure 3: Hypothetical example of a planning-focused metacognitive intervention built into ChatGPT. After the user specifies a task, the system automatically comes up with a decomposed, step-by-step guide for completion (left side of the figure). This could be aided by further proactive prompting, giving concrete examples of how sub-tasks could be solved (right side of the figure). An option to skip the decomposition step (bottom of the figure) minimises unnecessary cognitive load if decomposition is not required.
• Figure 4: Hypothetical example of a metacognitive intervention focused on self-evaluation built into Microsoft Copilot. In (A), the user provides a highly unspecified prompt to the system for writing a proposal. Based on a neutral assessment of similar prompting history, the GenAI system suggests reducing editing time by reflecting on more strategies. In (B) the user provides a highly unspecified prompt for summarizing. Based on a neutral assessment of previous interactions, the GenAI system suggests to limit interactions by suggesting to reflect on the user's more specific goals and intentions for this summary. These appear as suggestions next to the main chat window and can be closed if not wanted.
• Figure 5: Hypothetical example illustrating a metacognitive intervention focused on self-management and self-evaluation for coding in GitHub Copilot. During programming, the system can provide self-evaluation prompts to encourage user reflection on bugs and purpose of the code (right side of the figure). To decrease cognitive load if evaluation is not wanted, the user has the option to schedule evaluation for later, or set their own confidence level to increase or decrease the amount of suggestions (the 'proficiency settings' in the bottom right of the figure). The system could also prompt the user to think about the broader work context, for example whether this code-snippet may be relevant somewhere else in the overall code as well. The user has the options to ignore this suggestion, look at the other relevant areas now, or review them later (bottom left of the figure). To minimize cognitive load, the timing and frequency of prompts should be adapted to the users' preferences, expertise, and workflow.