Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

To Google or To ChatGPT? A Comparison of CS2 Students' Information Gathering Approaches and Outcomes

Aayush Kumar, Daniel Prol, Amin Alipour, Sruti Srinivasa Ragavan

TL;DR

This study addresses how CS2 students gather information when learning programming concepts with AI-based tools versus traditional online resources. Using a within-subject lab design at IIT Kanpur, participants learned two JavaScript concepts (currying and IIFE) under AI and NoAI conditions, with prompts/queries, screen actions, quizzes, and debugging tasks analyzed to compare strategies and outcomes. The findings show that while AI prompts generate more follow-ups, NoAI often yields stronger conceptual understanding, particularly for the harder currying task, and that learning with AI can hinder deep understanding if not paired with structured pedagogy and multimodal support. The work highlights the need for balanced, example-rich, and holistic AI-assisted learning designs that complement traditional resources rather than replace them, with implications for curriculum design, prompt engineering, and multimodal educational content.

Abstract

LLMs such as ChatGPT have been widely adopted by students in higher education as tools for learning programming and related concepts. However, it remains unclear how effective students are and what strategies students use while learning with LLMs. Since the majority of students' experiences in online self-learning have come through using search engines such as Google, evaluating AI tools in this context can help us address these gaps. In this mixed methods research, we conducted an exploratory within-subjects study to understand how CS2 students learn programming concepts using both LLMs as well as traditional online methods such as educational websites and videos to examine how students approach learning within and across both scenarios. We discovered that students found it easier to learn a more difficult concept using traditional methods than using ChatGPT. We also found that students ask fewer follow-ups and use more keyword-based queries for search engines while their prompts to LLMs tend to explicitly ask for information.

To Google or To ChatGPT? A Comparison of CS2 Students' Information Gathering Approaches and Outcomes

TL;DR

This study addresses how CS2 students gather information when learning programming concepts with AI-based tools versus traditional online resources. Using a within-subject lab design at IIT Kanpur, participants learned two JavaScript concepts (currying and IIFE) under AI and NoAI conditions, with prompts/queries, screen actions, quizzes, and debugging tasks analyzed to compare strategies and outcomes. The findings show that while AI prompts generate more follow-ups, NoAI often yields stronger conceptual understanding, particularly for the harder currying task, and that learning with AI can hinder deep understanding if not paired with structured pedagogy and multimodal support. The work highlights the need for balanced, example-rich, and holistic AI-assisted learning designs that complement traditional resources rather than replace them, with implications for curriculum design, prompt engineering, and multimodal educational content.

Abstract

LLMs such as ChatGPT have been widely adopted by students in higher education as tools for learning programming and related concepts. However, it remains unclear how effective students are and what strategies students use while learning with LLMs. Since the majority of students' experiences in online self-learning have come through using search engines such as Google, evaluating AI tools in this context can help us address these gaps. In this mixed methods research, we conducted an exploratory within-subjects study to understand how CS2 students learn programming concepts using both LLMs as well as traditional online methods such as educational websites and videos to examine how students approach learning within and across both scenarios. We discovered that students found it easier to learn a more difficult concept using traditional methods than using ChatGPT. We also found that students ask fewer follow-ups and use more keyword-based queries for search engines while their prompts to LLMs tend to explicitly ask for information.
Paper Structure (34 sections, 8 figures, 10 tables)

This paper contains 34 sections, 8 figures, 10 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Student-AI Interaction in Programming Education
  4. ChatGPT as a Learning Tool
  5. Online Information-Seeking Strategies in Programming
  6. AI and Web-Based Resources in Self-Directed Learning
  7. Research Questions
  8. Methodology
  9. Study Design
  10. Participants and recruitment
  11. Study Protocol
  12. Tasks and Assessments
  13. Tasks.
  14. Assessment.
  15. Data Collection and Analysis
  16. ...and 19 more sections

Figures (8)

  • Figure 1: Study Protocol (A/B refer to Currying or IIFE; X/Y refer to AI or NoAI)
  • Figure 2: Example application of activity analysis codebook (vertical divisions in a segment imply multiple codes applied)
  • Figure 3: Percentage of prompts and queries over time
  • Figure 4: Debugging Task Performance (LoU refers to lack of understanding)
  • Figure 6: IIFE Debugging Task - Problem Statement
  • ...and 3 more figures