Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Memento: Augmenting Personalized Memory via Practical Multimodal Wearable Sensing in Visual Search and Wayfinding Navigation

Indrajeet Ghosh, Kasthuri Jayarajah, Nicholas Waytowich, Nirmalya Roy

TL;DR

Memento addresses the challenge of augmenting short-term working memory during visual search and wayfinding by combining multimodal wearable sensing (EEG,GSR,PPG) with functional EEG signal reconstruction and in situ ERP cueing. The framework employs preprocessing, incremental multimodal fusion, and ERP-episode extraction via CPD and Morlet-wavelet CWT to detect high-attentional moments, which are then used to guide memory recall. In two studies, Memento improves route recall by 20–23%, reduces cognitive load and review time by 46%, and achieves ~75% faster processing than CV-based cueing, demonstrating practical, privacy-conscious memory augmentation in real-world navigation tasks. The work establishes a foundation for personalized, physiology-driven memory aids and highlights trade-offs between accuracy, noise reduction, and computation for edge-enabled in situ cues.

Abstract

Working memory involves the temporary retention of information over short periods. It is a critical cognitive function that enables humans to perform various online processing tasks, such as dialing a phone number, recalling misplaced items' locations, or navigating through a store. However, inherent limitations in an individual's capacity to retain information often result in forgetting important details during such tasks. Although previous research has successfully utilized wearable and assistive technologies to enhance long-term memory functions (e.g., episodic memory), their application to supporting short-term recall in daily activities remains underexplored. To address this gap, we present Memento, a framework that uses multimodal wearable sensor data to detect significant changes in cognitive state and provide intelligent in situ cues to enhance recall. Through two user studies involving 15 and 25 participants in visual search navigation tasks, we demonstrate that participants receiving visual cues from Memento achieved significantly better route recall, improving approximately 20-23% compared to free recall. Furthermore, Memento reduced cognitive load and review time by 46% while also substantially reducing computation time (3.86 seconds vs. 15.35 seconds), offering an average of 75% effectiveness compared to computer vision-based cue selection approaches.

Memento: Augmenting Personalized Memory via Practical Multimodal Wearable Sensing in Visual Search and Wayfinding Navigation

TL;DR

Memento addresses the challenge of augmenting short-term working memory during visual search and wayfinding by combining multimodal wearable sensing (EEG,GSR,PPG) with functional EEG signal reconstruction and in situ ERP cueing. The framework employs preprocessing, incremental multimodal fusion, and ERP-episode extraction via CPD and Morlet-wavelet CWT to detect high-attentional moments, which are then used to guide memory recall. In two studies, Memento improves route recall by 20–23%, reduces cognitive load and review time by 46%, and achieves ~75% faster processing than CV-based cueing, demonstrating practical, privacy-conscious memory augmentation in real-world navigation tasks. The work establishes a foundation for personalized, physiology-driven memory aids and highlights trade-offs between accuracy, noise reduction, and computation for edge-enabled in situ cues.

Abstract

Working memory involves the temporary retention of information over short periods. It is a critical cognitive function that enables humans to perform various online processing tasks, such as dialing a phone number, recalling misplaced items' locations, or navigating through a store. However, inherent limitations in an individual's capacity to retain information often result in forgetting important details during such tasks. Although previous research has successfully utilized wearable and assistive technologies to enhance long-term memory functions (e.g., episodic memory), their application to supporting short-term recall in daily activities remains underexplored. To address this gap, we present Memento, a framework that uses multimodal wearable sensor data to detect significant changes in cognitive state and provide intelligent in situ cues to enhance recall. Through two user studies involving 15 and 25 participants in visual search navigation tasks, we demonstrate that participants receiving visual cues from Memento achieved significantly better route recall, improving approximately 20-23% compared to free recall. Furthermore, Memento reduced cognitive load and review time by 46% while also substantially reducing computation time (3.86 seconds vs. 15.35 seconds), offering an average of 75% effectiveness compared to computer vision-based cue selection approaches.
Paper Structure (19 sections, 3 equations, 16 figures, 4 tables)

This paper contains 19 sections, 3 equations, 16 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Designing Memento
  4. Signal Pre-processing
  5. Multimodal Fusion
  6. Event-related Potentials (ERPs) Episodes Extraction
  7. Working Memory Dataset
  8. Study 1: High-attentional ERP Episodes Extraction via Physiological Signals
  9. Study Environments:
  10. Study Protocol:
  11. STUDY 2: Improving of Navigation Memory Recall through ERP Episodes Cueing
  12. Study Protocol:
  13. Results and Discussion
  14. RQ1: Extraction of High-Attentional ERPs
  15. RQ2: Improving Route Recall via ERPs
  16. ...and 4 more sections

Figures (16)

  • Figure 1: Overview of the Memento Framework
  • Figure 2: An example of preprocessed GSR signal highlighting SCR Peaks, Phasic Component, SCR Onsets, SCR Half Recovery phases and smooth, denoise/clean GSR signals, respectively.
  • Figure 3: Original EEG channel against the reconstructed signals.
  • Figure 4: Raw and cleaned PPG signals with peaks corresponding to heartbeats
  • Figure 6: Indoor
  • ...and 11 more figures