Evidence-centered Assessment for Writing with Generative AI

TL;DR

The paper tackles how to assess learning when students co-create writing with generative AI, addressing attribution of credit between human and AI actions. It develops an evidence-centered design framework that leverages ENA to model process-level differences in human–AI co-writing using the CoAuthor platform. Empirical results show meaningful differences driven by ownership and genre, supporting the viability of ENA-based process assessment for human–AI collaboration while acknowledging limitations of the CoAuthor dataset and GPT-3 era. The work offers a path toward scalable, adaptable assessments that capture the dynamics of AI-assisted writing in educational settings.

Abstract

We propose a learning analytics-based methodology for assessing the collaborative writing of humans and generative artificial intelligence. Framed by the evidence-centered design, we used elements of knowledge-telling, knowledge transformation, and cognitive presence to identify assessment claims; we used data collected from the CoAuthor writing tool as potential evidence for these claims; and we used epistemic network analysis to make inferences from the data about the claims. Our findings revealed significant differences in the writing processes of different groups of CoAuthor users, suggesting that our method is a plausible approach to assessing human-AI collaborative writing.

Figures (5)

• Figure 1: Methodology overview (the leftmost image sourced from Lee2022)
• Figure 2: ENA embedding space
• Figure 3: Ownership
• Figure 4: Prompt type
• Figure 5: Temperature