6D Pose Estimation on Spoons and Hands

TL;DR

This work addresses non-intrusive dietary monitoring by developing a $6$D pose estimation pipeline for hands and spoons during eating. The approach combines zero-shot segmentation (Grounding DINO + SAM2 or Cutie) with depth estimation (UniDepth) and $6$-DoF pose tracking (BundleSDF) to analyze eating actions in realistic video. Results show that SAM2-based segmentation yields higher accuracy and more robust pose tracking than Cutie, with BundleSDF maintaining consistent orientation during motion and depth changes, though segmentation quality remains a limiting factor. The framework demonstrates potential for automated estimation of food intake and eating behaviors, offering a path toward scalable, vision-based dietary analysis in naturalistic settings.

Abstract

Accurate dietary monitoring is essential for promoting healthier eating habits. A key area of research is how people interact and consume food using utensils and hands. By tracking their position and orientation, it is possible to estimate the volume of food being consumed, or monitor eating behaviours, highly useful insights into nutritional intake that can be more reliable than popular methods such as self-reporting. Hence, this paper implements a system that analyzes stationary video feed of people eating, using 6D pose estimation to track hand and spoon movements to capture spatial position and orientation. In doing so, we examine the performance of two state-of-the-art (SOTA) video object segmentation (VOS) models, both quantitatively and qualitatively, and identify main sources of error within the system.

Figures (9)

• Figure 1: Summary of pose estimation pipeline. Frames are extracted from videos and passed into Grounding DINO and SAM2 for zero-shot segmentation of spoons and hands. The initial segmented frame is passed into either Cutie or SAM2 for video object segmentation and the resulting 6D pose estimation is evaluated.
• Figure 2: Example of segmentation and pose estimation results using the Cutie and SAM2 models. First row displays results for a spoon. Second row displays results for the hand on the right.
• Figure 3: Inaccurate segmentations, marked in blue, from Cutie and SAM2 due to motion blur.
• Figure 4: 6D Pose estimates for the spoon during an eating motion produced from segmented masks by both Cutie and SAM2.
• Figure 5: BundleSDF is able to track objects despite poor segmentation.