Modifying AI, Enhancing Essays: How Active Engagement with Generative AI Boosts Writing Quality
Kaixun Yang, Mladen Raković, Zhiping Liang, Lixiang Yan, Zijie Zeng, Yizhou Fan, Dragan Gašević, Guanliang Chen
TL;DR
This paper addresses how students' GAI-assisted writing behaviors influence essay quality and proposes a causal framework to quantify these effects. It defines three treatments—T1 (seek suggestions -> not accept), T2 (seek suggestions -> accept without revision), and T3 (seek suggestions -> accept and revise)—and four outcomes (Y1: lexical sophistication, Y2: syntactic complexity, Y3: text cohesion, Y4: gender bias), estimating causal effects with $ATE = E[Y(1)] - E[Y(0)]$ and $ITE_i = Y_i(1) - Y_i(0)$ using the X-learner on the CoAuthor dataset. The main finding is that actively revising GAI suggestions (T3) consistently improves lexical sophistication, syntactic complexity, and text cohesion, while directly adopting AI text (T2) or only seeking suggestions (T1) can reduce these measures; all three approaches reduce gender bias to varying extents. These results suggest educators should emphasize meaningful in-process engagement with AI tools and develop in-process assessment methods to capture learning opportunities, while noting limitations due to dataset scope and variable availability.
Abstract
Students are increasingly relying on Generative AI (GAI) to support their writing-a key pedagogical practice in education. In GAI-assisted writing, students can delegate core cognitive tasks (e.g., generating ideas and turning them into sentences) to GAI while still producing high-quality essays. This creates new challenges for teachers in assessing and supporting student learning, as they often lack insight into whether students are engaging in meaningful cognitive processes during writing or how much of the essay's quality can be attributed to those processes. This study aimed to help teachers better assess and support student learning in GAI-assisted writing by examining how different writing behaviors, especially those indicative of meaningful learning versus those that are not, impact essay quality. Using a dataset of 1,445 GAI-assisted writing sessions, we applied the cutting-edge method, X-Learner, to quantify the causal impact of three GAI-assisted writing behavioral patterns (i.e., seeking suggestions but not accepting them, seeking suggestions and accepting them as they are, and seeking suggestions and accepting them with modification) on four measures of essay quality (i.e., lexical sophistication, syntactic complexity, text cohesion, and linguistic bias). Our analysis showed that writers who frequently modified GAI-generated text-suggesting active engagement in higher-order cognitive processes-consistently improved the quality of their essays in terms of lexical sophistication, syntactic complexity, and text cohesion. In contrast, those who often accepted GAI-generated text without changes, primarily engaging in lower-order processes, saw a decrease in essay quality. Additionally, while human writers tend to introduce linguistic bias when writing independently, incorporating GAI-generated text-even without modification-can help mitigate this bias.