Coach not crutch: Evidence that AI can improve writing skill despite reducing effort

TL;DR

It is suggested that AI can exert opposing effects on effort and learning rate -- making it possible in at least some cases to work less and learn more.

Abstract

In a series of highly-powered empirical studies, we examine the intuition that by sparing effort, using AI inevitably hinders learning. First, in a nationally representative survey of young adults, the majority expressed the view that using AI makes people lazier and less capable. Next, in a random-assignment experiment, we gave participants a tutorial on best practices in professional writing, then provided one group with access to an AI writing tool and asked another to practice writing on their own. Those who practiced with AI indeed exerted less effort while practicing -- yet wrote better cover letters in no-AI writing tests. In a second experiment with more rigorous control conditions, access to AI improved writing more than either googling cover letter examples and tips or receiving personalized feedback on their practice letters from experienced human editors. A third experiment explained these learning gains by showing that AI can teach by example: participants who viewed a cover letter that had been revised by the AI tool but did no further practice improved their writing as much as those who practiced writing with the original AI tool. Collectively, these pre-registered experiments suggest that AI can exert opposing effects on effort and learning rate -- making it possible in at least some cases to work less and learn more.

Figures (21)

• Figure 1: Conceptual framework. Relying on AI may cause people to exert less effort but improve the learning environment. Post-practice writing skill depends on the net effect of effort and the quality of the learning environment.
• Figure 2: The AI writing tool we created for this investigation mimics a widely used and publicly available AI writing tool that takes input text (left panel) and generates a version that incorporates recommended writing principles (right panel).
• Figure 3: Experimental design for Study 2. First, all participants completed a baseline questionnaire, a pretest (rewriting a poorly-written cover letter), and a lesson introducing five evidence-based principles of effective writing rogers2023. Next, participants were randomly assigned to one of three conditions: practicing with an AI writing tool, practicing without an AI writing tool, or no practice. Then, all participants were tested on writing skill (rewriting a new cover letter without access to AI) and completed an exit questionnaire. Finally, to assess the persistence of skill improvement, participants were invited to complete a similar incentivized test of writing skill one day later.
• Figure 4: Screenshot of one of the pages of the lesson shown to all participants.
• Figure 5: In a nationally representative sample of young adults, most endorsed the view that AI makes people less smart, lazier, and less likely to learn.