Untangling Critical Interaction with AI in Students Written Assessment
Antonette Shibani, Simon Knight, Kirsty Kitto, Ajanie Karunanayake, Simon Buckingham Shum
TL;DR
In the GenAI era, the paper investigates how learners interact critically with AI during writing and introduces the CIAW framework by integrating CPMW, IPS-I, and SAL to capture Deep, Shallow, or Absent engagement across five dimensions plus reflective and conversational aspects. A qualitative analysis of 49 graduate student submissions reveals predominantly shallow engagement with AI, underscoring the need for curricular and assessment redesign to foster mindful, evidence-based collaboration with AI. The study provides empirical groundwork and a coding scheme to guide tool design, scaffolding, and future research on enhancing AI literacy and metacognitive skills in writing tasks. The work highlights practical implications for educators and developers aiming to enable effective human-AI partnerships in higher education writing tasks.
Abstract
Artificial Intelligence (AI) has become a ubiquitous part of society, but a key challenge exists in ensuring that humans are equipped with the required critical thinking and AI literacy skills to interact with machines effectively by understanding their capabilities and limitations. These skills are particularly important for learners to develop in the age of generative AI where AI tools can demonstrate complex knowledge and ability previously thought to be uniquely human. To activate effective human-AI partnerships in writing, this paper provides a first step toward conceptualizing the notion of critical learner interaction with AI. Using both theoretical models and empirical data, our preliminary findings suggest a general lack of Deep interaction with AI during the writing process. We believe that the outcomes can lead to better task and tool design in the future for learners to develop deep, critical thinking when interacting with AI.