FIRE: Food Image to REcipe generation

TL;DR

FIRE introduces a multimodal pipeline that generates a complete recipe—title, ingredients, and cooking instructions—from a food image. It combines BLIP-based title generation, a ViT-derived ingredient extractor with a set-predictor, and a T5-based instruction generator, achieving state-of-the-art results on Recipe1M and enabling practical applications such as recipe customization and recipe-to-code generation via few-shot prompting. The approach demonstrates strong end-to-end performance and robust ingredient extraction, with ablations highlighting the benefits of ViT and cardinality regularization, while also uncovering challenges in grounding, generalization to novel dishes, and evaluation. The work offers a practical pathway for deploying image-to-recipe systems in real-world food computing, while outlining future directions to improve grounding, knowledge integration, and reliability of generated recipes.

Abstract

Food computing has emerged as a prominent multidisciplinary field of research in recent years. An ambitious goal of food computing is to develop end-to-end intelligent systems capable of autonomously producing recipe information for a food image. Current image-to-recipe methods are retrieval-based and their success depends heavily on the dataset size and diversity, as well as the quality of learned embeddings. Meanwhile, the emergence of powerful attention-based vision and language models presents a promising avenue for accurate and generalizable recipe generation, which has yet to be extensively explored. This paper proposes FIRE, a novel multimodal methodology tailored to recipe generation in the food computing domain, which generates the food title, ingredients, and cooking instructions based on input food images. FIRE leverages the BLIP model to generate titles, utilizes a Vision Transformer with a decoder for ingredient extraction, and employs the T5 model to generate recipes incorporating titles and ingredients as inputs. We showcase two practical applications that can benefit from integrating FIRE with large language model prompting: recipe customization to fit recipes to user preferences and recipe-to-code transformation to enable automated cooking processes. Our experimental findings validate the efficacy of our proposed approach, underscoring its potential for future advancements and widespread adoption in food computing.

Figures (5)

• Figure 1: Given a potentially unseen image, our method FIRE generates a corresponding recipe consisting of a title, ingredients, and cooking instructions.
• Figure 2: Proposed architecture to extract ingredients, and generate the recipe title and cooking instructions from a food image. (Ingredients with quantity is passed during the train time only)
• Figure 3: Generating cooking instructions for a title and a set of ingredients. (ingredients with quantity is present only during the fine-tuning of T5)
• Figure 4: Recipe prediction by FIRE for Pav Bhaji image.
• Figure 5: Applications of FIRE: Recipe Customization and Recipe to Code Generation.