Computer Science Achievement and Writing Skills Predict Vibe Coding Proficiency

Abstract

Many software development platforms now support LLM-driven programming, or "vibe coding", a technique that allows one to specify programs in natural language and iterate from observed behavior, all without directly editing source code. While its adoption is accelerating, little is known about which skills best predict success in this workflow. We report a preregistered cross-sectional study with tertiary-level students (N = 100) who completed measures of computer-science achievement, domain-general cognitive skills, written-communication proficiency, and a vibe-coding assessment. Tasks were curated via an eight-expert consensus process and executed in a purpose-built, vibe-coding environment that mirrors commercial tools while enabling controlled evaluation. We find that both writing skill and CS achievement are significant predictors of vibe-coding performance, and that CS achievement remains a significant predictor after controlling for domain-general cognitive skills. The results may inform tool and curriculum design, including when to emphasize prompt-writing versus CS fundamentals to support future software creators.

Figures (5)

• Figure 1: The grading rubric of written communication skills that we deployed in the study after expert consultation. Each sub-item is graded on a scale from 1 to 5, resulting in a maximum score of 20 for each category, and a maximum score of 100 in total.
• Figure 2: Items from assessment instruments ICAR16 (left) and SCS1 (right) that were provided as examples in the ICAR guidelines ICAR2014 and by Parker et al. parker2016replication, respectively.
• Figure 3: User interface of the tool we used in the study. The interface follows the vibe coding paradigm by not exposing any code. It features a chat box on the left, consistent with other tools, and an app preview on the right. The left sidebar also includes the task description and a timer showing the remaining time for participants (hovering over the bar shows the exact time remaining). The interaction seen in the screenshot was recreated based on prompts from participants during the study.
• Figure 4: While exploring the sample application, participants are provided with feedback about the number of relevant features they have interacted with.
• Figure 5: A diagram depicting the flow of our experiment. Participants were randomly divided into Group A and Group B. The order of the instruments in the test battery and the vibe coding tasks was randomized for each participant.