Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Designing and Evaluating Hint Generation Systems for Science Education

Anubhav Jangra, Smaranda Muresan

TL;DR

The paper investigates how automatic hint generation using open-source LLMs can support science education while mitigating answer leakage. It introduces a chain-of-hints framework with static and dynamic strategies and evaluates them through a within-subjects study with 41 participants on 30 open-ended SciQ questions, alongside a five-dimension automatic evaluation suite (information gain, redundancy, consistency, readability, leakage). The findings reveal that hints offer both benefits and challenges, with preferences varying by learner and task, and automatic metrics only partially reflecting learner experiences. The work provides design principles for learner-centered tutoring systems that promote self-regulated learning and careful integration of AI-based hints to avoid cognitive offloading.

Abstract

Large language models are influencing the education landscape, with students relying on them in their learning process. Often implemented using general-purpose models, these systems are likely to give away the answers, which could hinder conceptual understanding and critical thinking. We study the role of automatic hint generation as a pedagogical strategy to promote active engagement with the learning content, while guiding learners toward the answers. Focusing on scientific topics at the secondary education level, we explore the potential of large language models to generate chains of hints that scaffold learners without revealing answers. We compare two distinct hinting strategies: static hints, pre-generated for each problem, and dynamic hints, adapted to learners' progress. Through a quantitative study with 41 participants, we uncover different preferences among learners with respect to hinting strategies, and identify the limitations of automatic evaluation metrics to capture them. Our findings highlight key design considerations for future research on hint generation and intelligent tutoring systems that seek to develop learner-centered educational technologies.

Designing and Evaluating Hint Generation Systems for Science Education

TL;DR

The paper investigates how automatic hint generation using open-source LLMs can support science education while mitigating answer leakage. It introduces a chain-of-hints framework with static and dynamic strategies and evaluates them through a within-subjects study with 41 participants on 30 open-ended SciQ questions, alongside a five-dimension automatic evaluation suite (information gain, redundancy, consistency, readability, leakage). The findings reveal that hints offer both benefits and challenges, with preferences varying by learner and task, and automatic metrics only partially reflecting learner experiences. The work provides design principles for learner-centered tutoring systems that promote self-regulated learning and careful integration of AI-based hints to avoid cognitive offloading.

Abstract

Large language models are influencing the education landscape, with students relying on them in their learning process. Often implemented using general-purpose models, these systems are likely to give away the answers, which could hinder conceptual understanding and critical thinking. We study the role of automatic hint generation as a pedagogical strategy to promote active engagement with the learning content, while guiding learners toward the answers. Focusing on scientific topics at the secondary education level, we explore the potential of large language models to generate chains of hints that scaffold learners without revealing answers. We compare two distinct hinting strategies: static hints, pre-generated for each problem, and dynamic hints, adapted to learners' progress. Through a quantitative study with 41 participants, we uncover different preferences among learners with respect to hinting strategies, and identify the limitations of automatic evaluation metrics to capture them. Our findings highlight key design considerations for future research on hint generation and intelligent tutoring systems that seek to develop learner-centered educational technologies.
Paper Structure (28 sections, 8 equations, 14 figures, 3 tables)

This paper contains 28 sections, 8 equations, 14 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Automatic Hint Generation: Past and Present
  4. Automatic Evaluation of Hints
  5. Learner-centered evaluation of generated hints
  6. Scientific Hint Generation
  7. Problem Definition
  8. Dataset
  9. Baselines
  10. Automatic Evaluation Metrics
  11. User Study Design
  12. Findings
  13. When a Single Hint Helps (or Hurts)
  14. When One Hint Is Not Enough: The Dynamics of Chain-of-hints
  15. Static vs Dynamic hints
  16. ...and 13 more sections

Figures (14)

  • Figure 1: Illustration of automatic chain-of-hint generation for scientific question answering.
  • Figure 2: Performance of best models from five baseline model families evaluated using automatic evaluation metrics. Orange (solid) bars denote the static hint generation results and blue (striped) bars denote the dynamic hint generation results.
  • Figure 3: Our quiz interface for human evaluation study.
  • Figure 4: Statistics of 41 human evaluation study participants.
  • Figure 5: Interactions between the participants and static (left) and dynamic (right) hint generation strategies. The top interactions depict a successful journey, whereas the bottom interactions illustrate failure to get participants to reach the answer.
  • ...and 9 more figures