Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Understanding Student Interaction with AI-Powered Next-Step Hints: Strategies and Challenges

Anastasiia Birillo, Aleksei Rostovskii, Yaroslav Golubev, Hieke Keuning

TL;DR

The paper addresses how students interact with AI-powered next-step hints in an in-IDE Kotlin learning environment. It combines fine-grained task-solving logs from 34 students with 6 semi-structured interviews and applies process mining to identify 16 interaction scenarios, complemented by a public dataset. Key findings show distributions of hint interactions (e.g., 299 positive, 420 neutral, 284 negative) and reveal strategies like Selective Use of Hints and Combining Partial Solutions that students employ to cope with unhelpful hints. The work offers actionable insights for designing more effective hint systems and advances reproducible research by releasing the dataset for future study.

Abstract

Automated feedback generation plays a crucial role in enhancing personalized learning experiences in computer science education. Among different types of feedback, next-step hint feedback is particularly important, as it provides students with actionable steps to progress towards solving programming tasks. This study investigates how students interact with an AI-driven next-step hint system in an in-IDE learning environment. We gathered and analyzed a dataset from 34 students solving Kotlin tasks, containing detailed hint interaction logs. We applied process mining techniques and identified 16 common interaction scenarios. Semi-structured interviews with 6 students revealed strategies for managing unhelpful hints, such as adapting partial hints or modifying code to generate variations of the same hint. These findings, combined with our publicly available dataset, offer valuable opportunities for future research and provide key insights into student behavior, helping improve hint design for enhanced learning support.

Understanding Student Interaction with AI-Powered Next-Step Hints: Strategies and Challenges

TL;DR

The paper addresses how students interact with AI-powered next-step hints in an in-IDE Kotlin learning environment. It combines fine-grained task-solving logs from 34 students with 6 semi-structured interviews and applies process mining to identify 16 interaction scenarios, complemented by a public dataset. Key findings show distributions of hint interactions (e.g., 299 positive, 420 neutral, 284 negative) and reveal strategies like Selective Use of Hints and Combining Partial Solutions that students employ to cope with unhelpful hints. The work offers actionable insights for designing more effective hint systems and advances reproducible research by releasing the dataset for future study.

Abstract

Automated feedback generation plays a crucial role in enhancing personalized learning experiences in computer science education. Among different types of feedback, next-step hint feedback is particularly important, as it provides students with actionable steps to progress towards solving programming tasks. This study investigates how students interact with an AI-driven next-step hint system in an in-IDE learning environment. We gathered and analyzed a dataset from 34 students solving Kotlin tasks, containing detailed hint interaction logs. We applied process mining techniques and identified 16 common interaction scenarios. Semi-structured interviews with 6 students revealed strategies for managing unhelpful hints, such as adapting partial hints or modifying code to generate variations of the same hint. These findings, combined with our publicly available dataset, offer valuable opportunities for future research and provide key insights into student behavior, helping improve hint design for enhanced learning support.

Paper Structure

This paper contains 13 sections, 3 figures, 1 table.

Table of Contents

  1. Introduction
  2. Related Work & Background
  3. Prior Research on Hint Usage
  4. JetBrains Academy & Its AI-Based Hints
  5. Data Collection
  6. Data Collection Setup
  7. Task Solving Logs
  8. Interview Data
  9. Analysis & Results
  10. RQ1: Student Behavior Patterns
  11. RQ2: Student Strategies for Unhelpful Hints
  12. Threats to Validity
  13. Conclusion

Figures (3)

  • Figure 1: A process behavior graph showing hint request scenarios across all students. Large times (e.g., >900 seconds) indicate that the student was solving the task and then returned for another hint (i.e., a new scenario started).
  • Figure 2: An example of selective use of hints strategy: a student adapts the suggestion and applies it partially in their code. "Before changes" shows the student's initial code before requesting a hint. "Hints" shows both the text and code hints for the student's code. "After changes" shows the student's code after adapting and applying the hints.
  • Figure 3: An example of selective use of hints strategy: a student tries different combinations to find a correct one. "Hints" shows both the text and code hints for the student's code. "Student attempts" shows several attempts to apply the hints. The section below shows the final (correct) attempt by the student to apply the hints.