Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

ChefFusion: Multimodal Foundation Model Integrating Recipe and Food Image Generation

Peiyu Li, Xiaobao Huang, Yijun Tian, Nitesh V. Chawla

TL;DR

This work introduces a novel food computing foundation model that achieves true multimodality, encompassing tasks such as t2t, t2i, i2t, i2t, it2t, and t2ti.

Abstract

Significant work has been conducted in the domain of food computing, yet these studies typically focus on single tasks such as t2t (instruction generation from food titles and ingredients), i2t (recipe generation from food images), or t2i (food image generation from recipes). None of these approaches integrate all modalities simultaneously. To address this gap, we introduce a novel food computing foundation model that achieves true multimodality, encompassing tasks such as t2t, t2i, i2t, it2t, and t2ti. By leveraging large language models (LLMs) and pre-trained image encoder and decoder models, our model can perform a diverse array of food computing-related tasks, including food understanding, food recognition, recipe generation, and food image generation. Compared to previous models, our foundation model demonstrates a significantly broader range of capabilities and exhibits superior performance, particularly in food image generation and recipe generation tasks. We open-sourced ChefFusion at GitHub.

ChefFusion: Multimodal Foundation Model Integrating Recipe and Food Image Generation

TL;DR

This work introduces a novel food computing foundation model that achieves true multimodality, encompassing tasks such as t2t, t2i, i2t, i2t, it2t, and t2ti.

Abstract

Significant work has been conducted in the domain of food computing, yet these studies typically focus on single tasks such as t2t (instruction generation from food titles and ingredients), i2t (recipe generation from food images), or t2i (food image generation from recipes). None of these approaches integrate all modalities simultaneously. To address this gap, we introduce a novel food computing foundation model that achieves true multimodality, encompassing tasks such as t2t, t2i, i2t, it2t, and t2ti. By leveraging large language models (LLMs) and pre-trained image encoder and decoder models, our model can perform a diverse array of food computing-related tasks, including food understanding, food recognition, recipe generation, and food image generation. Compared to previous models, our foundation model demonstrates a significantly broader range of capabilities and exhibits superior performance, particularly in food image generation and recipe generation tasks. We open-sourced ChefFusion at GitHub.
Paper Structure (10 sections, 4 equations, 3 figures, 2 tables)

This paper contains 10 sections, 4 equations, 3 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Related work
  3. Methodology
  4. Training to Generate Recipe
  5. Training to Generate Food Image
  6. Dataset and Implement Details
  7. Experiments
  8. Evaluation Metrics
  9. Tasks
  10. Conclusion

Figures (3)

  • Figure 1: The architecture of ChefFusion: (1) Left: training the model to generate recipe by minimizing $l_{r}(x, y)$; (2) Right: training the model to generate food images by minimizing $l_{g}(y)$ and determine whether to produce text or images at each step by minimizing $l_{p}(y)$.
  • Figure 2: Inference procedure for ChefFusion: The model takes in image and text inputs, and generate text interleaved with food image.
  • Figure 3: Case Study: ChefFusion demonstrates a wide suite of multimodal capabilities, including food understanding, food recognition, recipe generation, food image generation and multimodal dialogue (left). Example of food images generated by ChefFusion (right).