An Integrated Framework for Contextual Personalized LLM-Based Food Recommendation

TL;DR

The work tackles the fragmentation and data-imbalance challenges in personalized Food-RecSys by proposing an integrated framework that synergizes domain-specific data components with Large Language Models. Central to the approach are the Multimedia Food Logger, the World Food Atlas, and the Food Recommendation as Language Processing (F-RLP) paradigm, which combines context generation, counterfactually retrained prompts, and query-stage personalization to deliver contextual, health-aware recommendations. Key contributions include a formal Personal Food Model (PFM) composed of Biological and Preferential components, an extendable data ecosystem (PFM, FKG, WFA), and a CFG-enabled LLM training loop that mitigates hallucinations and improves relevance. The framework demonstrates how location, context, and rich multimodal data can be integrated to produce precise, real-world food recommendations, with potential impacts on health outcomes and dietary behavior. Future work envisions broader data sources, automated logging, expanded knowledge graphs, and explainable, scalable LLM-based nutrition guidance.

Abstract

Personalized food recommendation systems (Food-RecSys) critically underperform due to fragmented component understanding and the failure of conventional machine learning with vast, imbalanced food data. While Large Language Models (LLMs) offer promise, current generic Recommendation as Language Processing (RLP) strategies lack the necessary specialization for the food domain's complexity. This thesis tackles these deficiencies by first identifying and analyzing the essential components for effective Food-RecSys. We introduce two key innovations: a multimedia food logging platform for rich contextual data acquisition and the World Food Atlas, enabling unique geolocation-based food analysis previously unavailable. Building on this foundation, we pioneer the Food Recommendation as Language Processing (F-RLP) framework - a novel, integrated approach specifically architected for the food domain. F-RLP leverages LLMs in a tailored manner, overcoming the limitations of generic models and providing a robust infrastructure for effective, contextual, and truly personalized food recommendations.

Figures (36)

• Figure 1: Structure of the thesis
• Figure 2: The food computing layers and the pyramid of the body of work. A simple search of publications in each layer by a few trivial keywords from each layer shows the focus of research tends to be more active in areas closer the centric layer demonstrating its more immediate importance in a human centric society.
• Figure 3: The interaction between the different layers of food computing shows that these layers are not isolated and will inevitably affect each other as they are interconnected. So a positive impact on all layer may start from the user layer's food intake choice based on preference and health.
• Figure 4: Overview of the Contextual Personal Food Recommendation Components. The red, orange, and green color indicating how established are the research field of each of these components in an individual level in the field of computer science. Showing the complexity of the problem even as individual components let alone as an integrated framework. Red: Requires intensive improvements, orange: fairly established, green: advanced level of establishment.
• Figure 5: Food logging will use multimedia input sources and complement information from online databases to log each meal and all metadata related to the meal. It captures information about the food (dish name, ingredients, quantity), location (place of eating), time (eating and logging), social context (companions), causal aspects (nutritional and flavor information), and multimedia and experiential information about the food.