DietGlance: Dietary Monitoring and Personalized Analysis at a Glance with Knowledge-Empowered AI Assistant

TL;DR

DietGlance introduces an eyewear-based system that automatically monitors dietary intake in daily life by detecting ingestive episodes with multimodal sensing and identifying foods from meal images using GPT-4V. To deliver reliable nutrition analysis and personalized guidance, it augments the LLM with a retrieval-augmented generation (RAG) module built on a curated nutrition library, addressing hallucination risks and data gaps. The authors validate the approach through a short-term study (N=33) with quantitative evaluations by crowd workers and domain experts, plus a four-week longitudinal study (N=16) showing improvements in dietary awareness, timing, and healthier choices. Results indicate DietGlance achieves high diet-identification performance, credible nutritional analysis, and beneficial long-term effects, with insights into privacy, personalization, and multi-device integration that inform future AI-assisted dietary monitoring and healthcare intervention systems.

Abstract

Growing awareness of wellness has prompted people to consider whether their dietary patterns align with their health and fitness goals. In response, researchers have introduced various wearable dietary monitoring systems and dietary assessment approaches. However, these solutions are either limited to identifying foods with simple ingredients or insufficient in providing analysis of individual dietary behaviors with domain-specific knowledge. In this paper, we present DietGlance, a system that automatically monitors dietary in daily routines and delivers personalized analysis from knowledge sources. DietGlance first detects ingestive episodes from multimodal inputs using eyeglasses, capturing privacy-preserving meal images of various dishes being consumed. Based on the inferred food items and consumed quantities from these images, DietGlance further provides nutritional analysis and personalized dietary suggestions, empowered by the retrieval augmentation generation module on a reliable nutrition library. A short-term user study (N=33) and a four-week longitudinal study (N=16) demonstrate the usability and effectiveness of DietGlance, offering insights and implications for future AI-assisted dietary monitoring and personalized healthcare intervention systems using eyewear.

Figures (10)

• Figure 1: The flow of DietGlance. (a) DietGlance uses Aria Glasses to detect ingestive episodes and collects data in real-world, unconstrained environments. Users can view their dietary analysis results using the (b) mobile (from left to right: meal log image presentation, detailed nutritional analysis, and personalized suggestions) or (c) desktop interface.
• Figure 2: System Overview.
• Figure 3: An illustration of the multimodal sensing framework for ingestive episode detection and diet image capture.
• Figure 4: The multimodal learning model for classifying ingestive episodes and others.
• Figure 5: An illustration of diet image capture and segment.