How AI Impacts Skill Formation

TL;DR

The study investigates whether AI-assisted coding aids or undermines skill formation when learners acquire a new library, focusing on the Trio asynchronous library. Using a randomized design with 52 professional or experienced programmers, the authors measure productivity and a 27-point conceptual/reading/debugging quiz across 7 Trio concepts. They find that AI assistance significantly impairs conceptual understanding, code reading, and debugging skills (about a 17% quiz score drop, $d=0.738$, $p=0.01$) with no robust average gains in task completion time; however, six AI interaction patterns emerge, of which three—prioritizing cognitive engagement (e.g., explanations, conceptual questions)—help preserve learning. The results imply that AI productivity is not a shortcut to competence and that careful integration of AI into workflows is needed to sustain skill formation, especially in safety-critical domains. The paper contributes a taxonomy of AI interaction patterns, detailed qualitative analysis, and an open data/transcripts resource to guide future research on human-AI collaboration in software engineering.

Abstract

AI assistance produces significant productivity gains across professional domains, particularly for novice workers. Yet how this assistance affects the development of skills required to effectively supervise AI remains unclear. Novice workers who rely heavily on AI to complete unfamiliar tasks may compromise their own skill acquisition in the process. We conduct randomized experiments to study how developers gained mastery of a new asynchronous programming library with and without the assistance of AI. We find that AI use impairs conceptual understanding, code reading, and debugging abilities, without delivering significant efficiency gains on average. Participants who fully delegated coding tasks showed some productivity improvements, but at the cost of learning the library. We identify six distinct AI interaction patterns, three of which involve cognitive engagement and preserve learning outcomes even when participants receive AI assistance. Our findings suggest that AI-enhanced productivity is not a shortcut to competence and AI assistance should be carefully adopted into workflows to preserve skill formation -- particularly in safety-critical domains.

Figures (27)

• Figure 1: Overview of results: (Left) We find a significant decrease in library-specific skills (conceptual understanding, code reading, and debugging) among workers using AI assistance for completing tasks with a new python library. (Right) We categorize AI usage patterns and found three high skill development patterns where participants stay cognitively engaged when using AI assistance.
• Figure 2: With AI assistance becoming more ubiquitous in the workplace, novice workers may complete tasks without the same learning outcomes. Our experiments aim to investigate the process of task completion requiring a new skill to understand the impact of AI assistance on coding skill formation.
• Figure 3: Experiment interface: We used a online interview platform to run our experiment. The treatment condition participants are prompted to use the AI assistant.
• Figure 4: Overview of learning task and comprehension check. All participants completed a warm-up coding task that did not require Trio knowledge. During the main Trio task, participants in the treatment group could use AI assistance to answer questions or generate code. All participants were not allowed to use AI in the comprehension check.
• Figure 5: Difference in means of overall task time and quiz score between the control (No AI) and treatment (AI Assistant) groups in Pilot Study D. Error bars represent 95% CI. Significance values correspond to treatment effect. * p<0.05, **<0.01, ***<0.001