HintEval: A Comprehensive Framework for Hint Generation and Evaluation for Questions

TL;DR

HintEval addresses fragmentation in hint generation and evaluation for QA by offering a Python-based, unified framework that integrates datasets, models, and metrics. It introduces a Dataset class, two built-in models (Answer-Aware and Answer-Agnostic), and a comprehensive set of evaluation metrics with multiple methods, enabling end-to-end hint generation and assessment across diverse datasets. The framework emphasizes reproducibility and accessibility through preprocessed datasets and extensive documentation, and is available on PyPI and GitHub. By supporting both answer-aware and answer-agnostic workflows and providing extensible interfaces for third-party models, HintEval aims to accelerate research and practical applications that promote critical thinking and problem-solving in NLP/IR.

Abstract

Large Language Models (LLMs) are transforming how people find information, and many users turn nowadays to chatbots to obtain answers to their questions. Despite the instant access to abundant information that LLMs offer, it is still important to promote critical thinking and problem-solving skills. Automatic hint generation is a new task that aims to support humans in answering questions by themselves by creating hints that guide users toward answers without directly revealing them. In this context, hint evaluation focuses on measuring the quality of hints, helping to improve the hint generation approaches. However, resources for hint research are currently spanning different formats and datasets, while the evaluation tools are missing or incompatible, making it hard for researchers to compare and test their models. To overcome these challenges, we introduce HintEval, a Python library that makes it easy to access diverse datasets and provides multiple approaches to generate and evaluate hints. HintEval aggregates the scattered resources into a single toolkit that supports a range of research goals and enables a clear, multi-faceted, and reliable evaluation. The proposed library also includes detailed online documentation, helping users quickly explore its features and get started. By reducing barriers to entry and encouraging consistent evaluation practices, HintEval offers a major step forward for facilitating hint generation and analysis research within the NLP/IR community.

Figures (6)

• Figure 1: HintEval logo.
• Figure 2: Example hints for a sample question with scoring metrics. The metrics Relevance, Convergence, Familiarity, and Answer Leakage are rated on a scale from 0 to 1, where 0 represents the lowest and 1 the highest value. Higher scores in Relevance, Convergence, and Familiarity indicate better results, while a lower score is preferable for Answer Leakage. The Readability metric is scored as 0 (Beginner), 1 (Intermediate), or 2 (Advanced), with lower values indicating better readability.
• Figure 3: Workflow of the HintEval: ① Questions are loaded and converted into a structured dataset using the Dataset module. ② Users can load preprocessed datasets as a structured dataset. ③ Hints can be generated for each question using the Model module and stored in the dataset object. ④ The Evaluation module assesses all generated hints and questions using various evaluation metrics, storing the results in the dataset object. ⑤ The updated dataset can be saved and reloaded as needed.
• Figure 4: A docstring for the evaluate function of the Wikipedia method within the Familiarity evaluation metric. The docstring begins with: ① A detailed description of the function, followed by ② Notes specific to the evaluation metric and the method. It includes ③ a comprehensive Example demonstrating usage, helping users understand how to effectively implement it. ④ The References section lists the scholarly publications referenced by the method, while the ⑤ See Also section provides links to related functions with similar functionality. ⑥ The Params section outlines the input parameters of the function, and ⑦ the Returns section specifies the expected output. This structure ensures clear, accessible, and thorough documentation for users.
• Figure 5: Schema of the Dataset class, illustrating the objects used to represent a dataset in HintEval. The arrows indicate a subclass relationship.