Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

"Real Learner Data Matters" Exploring the Design of LLM-Powered Question Generation for Deaf and Hard of Hearing Learners

Si Cheng, Shuxu Huffman, Qingxiaoyang Zhu, Haotian Su, Raja Kushalnagar, Qi Wang

TL;DR

The findings indicate that while LLMs offer significant potential for personalized learning, challenges remain in the interaction accessibility for the diverse DHH community.

Abstract

Deaf and Hard of Hearing (DHH) learners face unique challenges in learning environments, often due to a lack of tailored educational materials that address their specific needs. This study explores the potential of Large Language Models (LLMs) to generate personalized quiz questions to enhance DHH students' video-based learning experiences. We developed a prototype leveraging LLMs to generate questions with emphasis on two unique strategies: Visual Questions, which identify video segments where visual information might be misrepresented, and Emotion Questions, which highlight moments where previous DHH learners experienced learning difficulty manifested in emotional responses. Through user studies with DHH undergraduates, we evaluated the effectiveness of these LLM-generated questions in supporting the learning experience. Our findings indicate that while LLMs offer significant potential for personalized learning, challenges remain in the interaction accessibility for the diverse DHH community. The study highlights the importance of considering language diversity and culture in LLM-based educational technology design.

"Real Learner Data Matters" Exploring the Design of LLM-Powered Question Generation for Deaf and Hard of Hearing Learners

TL;DR

The findings indicate that while LLMs offer significant potential for personalized learning, challenges remain in the interaction accessibility for the diverse DHH community.

Abstract

Deaf and Hard of Hearing (DHH) learners face unique challenges in learning environments, often due to a lack of tailored educational materials that address their specific needs. This study explores the potential of Large Language Models (LLMs) to generate personalized quiz questions to enhance DHH students' video-based learning experiences. We developed a prototype leveraging LLMs to generate questions with emphasis on two unique strategies: Visual Questions, which identify video segments where visual information might be misrepresented, and Emotion Questions, which highlight moments where previous DHH learners experienced learning difficulty manifested in emotional responses. Through user studies with DHH undergraduates, we evaluated the effectiveness of these LLM-generated questions in supporting the learning experience. Our findings indicate that while LLMs offer significant potential for personalized learning, challenges remain in the interaction accessibility for the diverse DHH community. The study highlights the importance of considering language diversity and culture in LLM-based educational technology design.
Paper Structure (34 sections, 8 figures)

This paper contains 34 sections, 8 figures.

Table of Contents

  1. Introduction
  2. Related Works
  3. DHH Learning with Video-Based Materials
  4. DHH Learning with Emerging Technology
  5. Automatic Language Generation for Learning Materials
  6. Methods
  7. Prototype Design
  8. Interactive LLM Chatbot for Question Personalization
  9. Video Display with Question Pop-Up and Feedback
  10. User Study (N=16)
  11. Onboarding
  12. Step 1: Chat with Chatbot to Personalize Question Set (Fig. \ref{['fig:system_screens']} - A)
  13. Step 2: Watch Video and Answer Pop-up Personalize Question Set (Fig. \ref{['fig:system_screens']}- B & C & D & E)
  14. Step 3: Post-Watching - Evaluate Personalize Quiz Question Set
  15. Step 4: User Experience
  16. ...and 19 more sections

Figures (8)

  • Figure 1: The research team uses ChatGPT API to process video transcripts to generate quiz questions using three different strategies. The first strategy involves using only the transcript. The second adds emotional data at specific timestamps, and the third includes visual information from the video at corresponding points. Each strategy generates 10 candidate questions stored in a question bank, and students can choose one to three strategies to create a set of 10 questions they will answer in user study.
  • Figure 2: Above, the chatbot greets participants by explaining the main task, which includes generating quiz questions using three different strategies, and prompts them to ask for clarification. Participants are encouraged to engage in conversation with the chatbot. When a participant types "hello," the chatbot redirects the conversation to focus on the questions generated with three strategies (A). Below and to the left, after the participant confirms 'emotion-enhanced transcript,', the video player displays the instructional video (B). To the right, the current question index number, the number of remaining questions for the video, the strategy on which the current question is based (C), the question itself (D), and the reference start timestamp (E) are indicated. There are three possible question-generating strategies based on the user’s interaction with the chatbot (A): Base transcript, emotion-enhanced transcript, and visual-enhanced transcript.
  • Figure 3: User Study Process
  • Figure 4: DHH participants selected one, two, or all three question generation strategies in step 1 and evaluated their selected types after watching the video. Qualitatively, our two proposed question generation strategies (Emotion and Visual) seem to outscore the Base transcript strategy in recalling facts/concepts. Emotion strategy scored higher for applying knowledge to new situations. Questions generated from the Base transcript received higher scores for fostering a connection between text and image.
  • Figure 5: Distribution of time to answer for different types of questions and the answering correct rates at the first attempts. After excluding potential outlier responses where participants took very long time to answer the questions, the results showed that the Base transcript questions took the shortest time to answer and received the highest correct rate, the Emotion Questions took the longest to answer and received an intermediate correct rate, and the Visual Questions took an intermediate time to answer with the lowest correct rate. The differences suggested that the Emotion and Visual Questions might identify video contents not fully understood by the participants, leading to longer answering time and lower correct rate. The Emotion Questions might elicit more thinking than the Visual Questions did, leading to a longer answering time and a higher correct rate.
  • ...and 3 more figures