Feel the Bite: Robot-Assisted Inside-Mouth Bite Transfer using Robust Mouth Perception and Physical Interaction-Aware Control

TL;DR

An inside-mouth bite transfer system that addresses challenges with two key components: a multi-view mouth perception pipeline robust to tool occlusion, and a control mechanism that employs multimodal time-series classification to discern and react to different physical interactions.

Abstract

Robot-assisted feeding can greatly enhance the lives of those with mobility limitations. Modern feeding systems can pick up and position food in front of a care recipient's mouth for a bite. However, many with severe mobility constraints cannot lean forward and need direct inside-mouth food placement. This demands precision, especially for those with restricted mouth openings, and appropriately reacting to various physical interactions - incidental contacts as the utensil moves inside, impulsive contacts due to sudden muscle spasms, deliberate tongue maneuvers by the person being fed to guide the utensil, and intentional bites. In this paper, we propose an inside-mouth bite transfer system that addresses these challenges with two key components: a multi-view mouth perception pipeline robust to tool occlusion, and a control mechanism that employs multimodal time-series classification to discern and react to different physical interactions. We demonstrate the efficacy of these individual components through two ablation studies. In a full system evaluation, our system successfully fed 13 care recipients with diverse mobility challenges. Participants consistently emphasized the comfort and safety of our inside-mouth bite transfer system, and gave it high technology acceptance ratings - underscoring its transformative potential in real-world scenarios. Supplementary materials and videos can be found at http://emprise.cs.cornell.edu/bitetransfer/ .

Figures (6)

• Figure 1: The self-supervised multi-view training of FEMTO (left) enables robust mouth perception under tool occlusion (center). Our perception pipeline (right) leverages FEMTO to accurately estimate mouth pose despite noisy depth due to occlusion.
• Figure 2: RealTime (Ours) enables the robot to track the mouth's pose and state, which enhances satisfaction, safety, comfort, confidence, control, ease of use, bite timing, and responsiveness across various scenarios in comparison to OneTimeshaikewitz2023mouth. Response options used across studies: SD (Strongly Disagree), D (Disagree), N (Neutral), A (Agree), SA (Strongly Agree).
• Figure 3: Confusion matrix showing multimodal SVM's performance for Condition B (Novel Participant).
• Figure 4: More physical-interaction aware control enhances perceived safety, satisfaction and comfort. This is supported by quantitative data recorded by the force sensor.
• Figure 5: SVM's classification performance on novel participants improves steadily with finetuning using their data.