Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

One Step at a Time: Combining LLMs and Static Analysis to Generate Next-Step Hints for Programming Tasks

Anastasiia Birillo, Elizaveta Artser, Anna Potriasaeva, Ilya Vlasov, Katsiaryna Dzialets, Yaroslav Golubev, Igor Gerasimov, Hieke Keuning, Timofey Bryksin

TL;DR

The results show that both forms of the hints — textual and code — were helpful for the students, and the proposed system helped them to proceed with the coding tasks.

Abstract

Students often struggle with solving programming problems when learning to code, especially when they have to do it online, with one of the most common disadvantages of working online being the lack of personalized help. This help can be provided as next-step hint generation, i.e., showing a student what specific small step they need to do next to get to the correct solution. There are many ways to generate such hints, with large language models (LLMs) being among the most actively studied right now. While LLMs constitute a promising technology for providing personalized help, combining them with other techniques, such as static analysis, can significantly improve the output quality. In this work, we utilize this idea and propose a novel system to provide both textual and code hints for programming tasks. The pipeline of the proposed approach uses a chain-of-thought prompting technique and consists of three distinct steps: (1) generating subgoals - a list of actions to proceed with the task from the current student's solution, (2) generating the code to achieve the next subgoal, and (3) generating the text to describe this needed action. During the second step, we apply static analysis to the generated code to control its size and quality. The tool is implemented as a modification to the open-source JetBrains Academy plugin, supporting students in their in-IDE courses. To evaluate our approach, we propose a list of criteria for all steps in our pipeline and conduct two rounds of expert validation. Finally, we evaluate the next-step hints in a classroom with 14 students from two universities. Our results show that both forms of the hints - textual and code - were helpful for the students, and the proposed system helped them to proceed with the coding tasks.

One Step at a Time: Combining LLMs and Static Analysis to Generate Next-Step Hints for Programming Tasks

TL;DR

The results show that both forms of the hints — textual and code — were helpful for the students, and the proposed system helped them to proceed with the coding tasks.

Abstract

Students often struggle with solving programming problems when learning to code, especially when they have to do it online, with one of the most common disadvantages of working online being the lack of personalized help. This help can be provided as next-step hint generation, i.e., showing a student what specific small step they need to do next to get to the correct solution. There are many ways to generate such hints, with large language models (LLMs) being among the most actively studied right now. While LLMs constitute a promising technology for providing personalized help, combining them with other techniques, such as static analysis, can significantly improve the output quality. In this work, we utilize this idea and propose a novel system to provide both textual and code hints for programming tasks. The pipeline of the proposed approach uses a chain-of-thought prompting technique and consists of three distinct steps: (1) generating subgoals - a list of actions to proceed with the task from the current student's solution, (2) generating the code to achieve the next subgoal, and (3) generating the text to describe this needed action. During the second step, we apply static analysis to the generated code to control its size and quality. The tool is implemented as a modification to the open-source JetBrains Academy plugin, supporting students in their in-IDE courses. To evaluate our approach, we propose a list of criteria for all steps in our pipeline and conduct two rounds of expert validation. Finally, we evaluate the next-step hints in a classroom with 14 students from two universities. Our results show that both forms of the hints - textual and code - were helpful for the students, and the proposed system helped them to proceed with the coding tasks.

Paper Structure

This paper contains 36 sections, 5 figures.

Table of Contents

  1. Introduction
  2. Background
  3. Personalized Feedback
  4. In-IDE Learning
  5. The Next-Step Hint System: Student's Perspective
  6. The Next-Step Hint System: Internal Design
  7. General Pipeline
  8. Generating Subgoals
  9. Generating Code and Textual Hints
  10. Code hint prompt
  11. Code hint quality improvement
  12. Textual hint prompt
  13. Internal Validation
  14. Overview
  15. UX Design of The Next-Step Hint System
  16. ...and 21 more sections

Figures (5)

  • Figure 1: The UI of the JetBrains Academy plugin together with an example of the developed next-step hint: (1) the "Get hint" button, (2) textual hint, and (3) code hint.
  • Figure 2: The overall pipeline of the next-step hint system.
  • Figure 3: Illustration of the impact of applying heuristics to reduce changes in student code. The green highlighting indicates the changes that were kept for the code hint. The red highlighting indicates changes that were removed by the heuristic.
  • Figure 4: Proposed UX options. Prototype A: a textual hint is shown in the Task description panel (1). Prototype B: a textual hint is shown in the Task description panel, and the line is highlighted where changes have to be made (1 + 2). Prototype C: a textual hint is shown in the code editor and points to the location where changes have to be made (3).
  • Figure 5: Example problems in the hint: (1) Unintended code modification, (2) Premature disclosure of the full solution, (3) Unwanted procedural comments in code, (4) Excessive detail in texual hint, (5) Limited recognition of built-in Kotlin functions.