Auditing Student-AI Collaboration: A Case Study of Online Graduate CS Students

TL;DR

This study empirically examines graduate CS students' collaboration with generative AI by mapping desired levels of automation against actual usage across 12 tasks using the Human Agency Scale. It integrates quantitative task-level data with qualitative design input to delineate four automation zones and to identify system features that would bolster trust, such as transparency, source citations, and explicit uncertainty signals. The findings reveal a task-dependent pattern: students want substantial automation for routine tasks but remain cautious for ideation and high-stakes reasoning, underscoring the need for calibrated AI assistance that preserves human agency. The work offers design guidelines and policy implications for creating trustworthy, human-centered educational AI that supports learning while mitigating risks like hallucinations and over-reliance.

Abstract

As generative AI becomes embedded in higher education, it increasingly shapes how students complete academic tasks. While these systems offer efficiency and support, concerns persist regarding over-automation, diminished student agency, and the potential for unreliable or hallucinated outputs. This study conducts a mixed-methods audit of student-AI collaboration preferences by examining the alignment between current AI capabilities and students' desired levels of automation in academic work. Using two sequential and complementary surveys, we capture students' perceived benefits, risks, and preferred boundaries when using AI. The first survey employs an existing task-based framework to assess preferences for and actual usage of AI across 12 academic tasks, alongside primary concerns and reasons for use. The second survey, informed by the first, explores how AI systems could be designed to address these concerns through open-ended questions. This study aims to identify gaps between existing AI affordances and students' normative expectations of collaboration, informing the development of more effective and trustworthy AI systems for education.

Figures (6)

• Figure 1: Illustration of the Likert-scale question used to measure students' desired level of AI automation for academic tasks.
• Figure 2: Illustration of the Likert-scale question used to capture students' self-reported observed AI usage across academic tasks.
• Figure 3: Comparison of students’ desired AI automation levels and self-reported actual AI usage across twelve academic tasks (T1–T12).
• Figure 4: Four-zone automation alignment map plotting desired AI automation against actual AI usage for twelve academic tasks.
• Figure 5: Distribution of students’ reported reasons for using AI across twelve academic tasks.