"It's not like Jarvis, but it's pretty close!" -- Examining ChatGPT's Usage among Undergraduate Students in Computer Science
Ishika Joshi, Ritvik Budhiraja, Harshal D Akolekar, Jagat Sesh Challa, Dhruv Kumar
TL;DR
The paper investigates how undergraduate computer science students in India actually use ChatGPT, addressing a gap in student-centered evaluation of LLMs in education. Using a mixed-methods approach with 480 surveys and 17 interviews across multiple Indian universities, it shows that over half of students view ChatGPT as beneficial for coursework, particularly for information gathering, summarization, coding assistance, and content creation. However, significant challenges persist, including unreliable outputs, difficulty verifying information, lack of sources, usability barriers, and ethical concerns about reliance and academic integrity. The study highlights opportunities to enhance LLM-based learning tools through confidence scores, provenance, domain-specific variants, prompting training, and streamlined workflows, with implications for instructors, tool designers, and future research in technology-assisted computing education.
Abstract
Large language models (LLMs) such as ChatGPT and Google Bard have garnered significant attention in the academic community. Previous research has evaluated these LLMs for various applications such as generating programming exercises and solutions. However, these evaluations have predominantly been conducted by instructors and researchers, not considering the actual usage of LLMs by students. This study adopts a student-first approach to comprehensively understand how undergraduate computer science students utilize ChatGPT, a popular LLM, released by OpenAI. We employ a combination of student surveys and interviews to obtain valuable insights into the benefits, challenges, and suggested improvements related to ChatGPT. Our findings suggest that a majority of students (over 57%) have a convincingly positive outlook towards adopting ChatGPT as an aid in coursework-related tasks. However, our research also highlights various challenges that must be resolved for long-term acceptance of ChatGPT amongst students. The findings from this investigation have broader implications and may be applicable to other LLMs and their role in computing education.