Assist-As-Needed: Adaptive Multimodal Robotic Assistance for Medication Management in Dementia Care

TL;DR

The paper tackles the challenge of enabling autonomous medication management for people with dementia by addressing the limitations of fixed, non-adaptive robotic assistance. It presents an adaptive multimodal framework implemented on the Pepper robot that escalates support from verbal reminders to step-by-step, multimodal guidance based on real-time assessment of user needs, guided by occupational therapy principles. The system integrates perception (YOLOv11 for object detection, depth-based 3D pose estimation), navigation (ROS-based 2D SLAM and ROI-driven guidance), and dialogue (GPT‑4o) with gestural cues and gaze grounding to preserve autonomy while ensuring adherence. Formative results with healthy adults and dementia-care stakeholders indicate high usability, faster task completion with richer cues, and clear design considerations for future deployment with PLWDs in more naturalistic settings, highlighting the framework's potential to reduce caregiver burden and improve independence in dementia care.

Abstract

People living with dementia (PLWDs) face progressively declining abilities in medication management-from simple forgetfulness to complete task breakdown-yet most assistive technologies fail to adapt to these changing needs. This one-size-fits-all approach undermines autonomy, accelerates dependence, and increases caregiver burden. Occupational therapy principles emphasize matching assistance levels to individual capabilities: minimal reminders for those who merely forget, spatial guidance for those who misplace items, and comprehensive multimodal support for those requiring step-by-step instruction. However, existing robotic systems lack this adaptive, graduated response framework essential for maintaining PLWD independence. We present an adaptive multimodal robotic framework using the Pepper robot that dynamically adjusts assistance based on real-time assessment of user needs. Our system implements a hierarchical intervention model progressing from (1) simple verbal reminders, to (2) verbal + gestural cues, to (3) full multimodal guidance combining physical navigation to medication locations with step-by-step verbal and gestural instructions. Powered by LLM-driven interaction strategies and multimodal sensing, the system continuously evaluates task states to provide just-enough assistance-preserving autonomy while ensuring medication adherence. We conducted a preliminary study with healthy adults and dementia care stakeholders in a controlled lab setting, evaluating the system's usability, comprehensibility, and appropriateness of adaptive feedback mechanisms. This work contributes: (1) a theoretically grounded adaptive assistance framework translating occupational therapy principles into HRI design, (2) a multimodal robotic implementation that preserves PLWD dignity through graduated support, and (3) empirical insights into stakeholder perceptions of adaptive robotic care.

Figures (8)

• Figure 1: Adaptive Assist-As-Needed for people with dementia, involving multimodal perception and feedback
• Figure 2: Overview of the framework for medication management. The arrows indicate the flow of information.
• Figure 3: Hardware components mounted on Pepper robot
• Figure 4: Experimental procedure
• Figure 5: The 2D occupancy grid map represents the lab used for the tests. Image on top left shows the interaction between the participant and the robot. Image on top right shows the interaction between the participant and the robot after it guided and detected the pill bottle.