Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Re:Member: Emotional Question Generation from Personal Memories

Zackary Rackauckas, Nobuaki Minematsu, Julia Hirschberg

TL;DR

Re:Member tackles the problem of generic, de-contextualized language-learning tools by leveraging learners' personal memory videos to generate emotionally expressive questions in the target language. It presents an end-to-end multimodal pipeline that uses Silero VAD and WhisperX for audio-visual alignment, a three-frame visual context, GPT-4o for context-aware question generation, a fixed set of Japanese emotion styles, and StyleBERT-VITS2 for expressive TTS. The key contributions include an open-source, memory-grounded framework and illustrative outputs that demonstrate context-sensitive, affective language prompts tied to real memories. The work highlights the potential of affective computing and memory-centered interaction to improve learner engagement and retention in L2 education, while outlining practical considerations for privacy and future enhancements.

Abstract

We present Re:Member, a system that explores how emotionally expressive, memory-grounded interaction can support more engaging second language (L2) learning. By drawing on users' personal videos and generating stylized spoken questions in the target language, Re:Member is designed to encourage affective recall and conversational engagement. The system aligns emotional tone with visual context, using expressive speech styles such as whispers or late-night tones to evoke specific moods. It combines WhisperX-based transcript alignment, 3-frame visual sampling, and Style-BERT-VITS2 for emotional synthesis within a modular generation pipeline. Designed as a stylized interaction probe, Re:Member highlights the role of affect and personal media in learner-centered educational technologies.

Re:Member: Emotional Question Generation from Personal Memories

TL;DR

Re:Member tackles the problem of generic, de-contextualized language-learning tools by leveraging learners' personal memory videos to generate emotionally expressive questions in the target language. It presents an end-to-end multimodal pipeline that uses Silero VAD and WhisperX for audio-visual alignment, a three-frame visual context, GPT-4o for context-aware question generation, a fixed set of Japanese emotion styles, and StyleBERT-VITS2 for expressive TTS. The key contributions include an open-source, memory-grounded framework and illustrative outputs that demonstrate context-sensitive, affective language prompts tied to real memories. The work highlights the potential of affective computing and memory-centered interaction to improve learner engagement and retention in L2 education, while outlining practical considerations for privacy and future enhancements.

Abstract

We present Re:Member, a system that explores how emotionally expressive, memory-grounded interaction can support more engaging second language (L2) learning. By drawing on users' personal videos and generating stylized spoken questions in the target language, Re:Member is designed to encourage affective recall and conversational engagement. The system aligns emotional tone with visual context, using expressive speech styles such as whispers or late-night tones to evoke specific moods. It combines WhisperX-based transcript alignment, 3-frame visual sampling, and Style-BERT-VITS2 for emotional synthesis within a modular generation pipeline. Designed as a stylized interaction probe, Re:Member highlights the role of affect and personal media in learner-centered educational technologies.
Paper Structure (13 sections, 1 figure)

This paper contains 13 sections, 1 figure.

Table of Contents

  1. Introduction
  2. System Overview
  3. Audio-Visual Segmentation
  4. Frame Sampling and Visual Context
  5. Multimodal Question Generation
  6. Emotion Style Selection
  7. Expressive Speech Synthesis
  8. Interactive User Interface
  9. Illustrative Outputs
  10. Discussion and Future Work
  11. Appendix A
  12. Question Generation Prompt
  13. Emotion Selection Prompt

Figures (1)

  • Figure 1: Example interface frame from video (1), showing (from top to bottom) three frames of sequential visual context, the generated emotion, the generated system question, a playable text-to-speech box, the name of the video file, the "Analyze Video" button, and "Previous" and "Next" buttons to navigate between sequential moments.