The Agency Gap: How Generative AI Literacy Shapes Independent Writing after AI Support

TL;DR

This study asks whether GenAI literacy ($GLAT$) translates into independent multimodal academic writing once AI support is removed, and whether reactive versus proactive chatbot strategies modulate this transfer. In a two-phase experiment with 79 medical and nursing students, GLAT and domain knowledge ($VLAT$) were measured alongside a five-dimension writing rubric; analyses included ordinal logistic regression and bootstrapped mediation. Results show GLAT predicts higher post-support writing only in the reactive condition, with no significant mediation through in-task performance, and proactive scaffolding equalises outcomes across literacy levels. The findings reveal an agency gap where GenAI literacy matters most when learners must self-initiate, informing the design of equitable AI-supported learning environments and curriculum integration of GenAI literacies.

Abstract

Generative AI (GenAI) tools are rapidly transforming higher education, yet little is known about how students' GenAI literacy shapes their ability to perform independently once such support is removed. This study investigates what we term the agency gap, introduced as the extent to which GenAI literacy predicts student writing performance in contexts that require self-initiation and regulation. Seventy-nine medical and nursing students completed multimodal academic writing tasks based on visual data, supported either by a reactive or proactive GenAI chatbot, followed by a parallel task without AI support. Writing was evaluated across insightfulness, visual data integration, organisation, linguistic quality, and critical thinking. Results showed that GenAI literacy predicted independent writing performance only in the reactive condition, where students had to actively mobilise their own strategies. Mediation analyses revealed no indirect effect via in-task performance, indicating that GenAI literacy acts as a direct, task-general competence rather than a proxy for domain knowledge or other literacies. By contrast, proactive scaffolding equalised outcomes across literacy levels, reducing reliance on learners' GenAI literacy. The agency gap highlights when GenAI literacy matters most, with implications for designing equitable AI-supported learning environments that both leverage and mitigate differences in students' GenAI literacy.

Figures (7)

• Figure 1: Three visuals showing (i) team behaviours (bar chart), (ii) team communication patterns (network diagram), and (iii) movement, communication intensity, and physiological responses (ward heatmap).
• Figure 2: Research platform showing (i) visual information display, (ii) academic writing space, and (iii) either AI-supported chat function (reactive or proactive) or (iv) standard task instruction (AI-removal condition).
• Figure 3: GenAI chatbot interaction design contrasting a reactive chatbot (red; right) responding only upon learner query and a proactive chatbot (blue; left) scaffolding learning through structured questioning and iterative feedback.
• Figure 4: Experimental procedure consisting of two phases: (1) AI-supported writing task with either reactive or proactive chatbot assistance, and (2) AI-removal writing task without chatbot support.
• Figure 5: Correlation between visualisation literacy and rubric metrics. * $\textit{p} < 0.05$, ** $\textit{p} < 0.01$, *** $\textit{p} < 0.001$