How Does India Cook Biryani?

TL;DR

This work addresses the challenge of capturing fine‑grained, culturally grounded procedural differences in Indian biryani across regions. It introduces a large, curated 120‑video biryani dataset and a multi‑stage vision–language model pipeline for temporal segmentation, multimodal alignment, and cross‑variant comparison, complemented by a hierarchical video question‑answering benchmark. The authors demonstrate robust segmentation and alignment, and propose a cross‑recipe comparison framework that reveals meaningful regional differences while highlighting areas for model improvement. The dataset, methods, and QA benchmarks enable interpretable, cross‑cultural, multimodal reasoning and open avenues for skill‑based video retrieval and heritage preservation, with release of data and code to the community.

Abstract

Biryani, one of India's most celebrated dishes, exhibits remarkable regional diversity in its preparation, ingredients, and presentation. With the growing availability of online cooking videos, there is unprecedented potential to study such culinary variations using computational tools systematically. However, existing video understanding methods fail to capture the fine-grained, multimodal, and culturally grounded differences in procedural cooking videos. This work presents the first large-scale, curated dataset of biryani preparation videos, comprising 120 high-quality YouTube recordings across 12 distinct regional styles. We propose a multi-stage framework leveraging recent advances in vision-language models (VLMs) to segment videos into fine-grained procedural units and align them with audio transcripts and canonical recipe text. Building on these aligned representations, we introduce a video comparison pipeline that automatically identifies and explains procedural differences between regional variants. We construct a comprehensive question-answer (QA) benchmark spanning multiple reasoning levels to evaluate procedural understanding in VLMs. Our approach employs multiple VLMs in complementary roles, incorporates human-in-the-loop verification for high-precision tasks, and benchmarks several state-of-the-art models under zero-shot and fine-tuned settings. The resulting dataset, comparison methodology, and QA benchmark provide a new testbed for evaluating VLMs on structured, multimodal reasoning tasks and open new directions for computational analysis of cultural heritage through cooking videos. We release all data, code, and the project website at https://farzanashaju.github.io/how-does-india-cook-biryani/.

Figures (22)

• Figure 1: Map of India showing 12 regional biryani types - Ambur, Bombay, Dindigul, Donne, Hyderabadi, Kashmiri, Kolkata, Awadhi, Malabar, Mughlai, Sindhi, and Thalassery. Representative images illustrate differences in preparation, ingredients, and presentation, with all videos sourced from YouTube to capture authentic regional cooking practices.
• Figure 2: Overview of the Biryani Dataset. Panels (a–d) show representative frames from four of the twelve biryani categories - Ambur, Dindigul, Hyderabadi, and Mughlai - capturing regional diversity in presentation, colour palette, and plating. Panels (e) and (f) present verb and noun frequency word clouds derived from ASR-transcribed and translated speech, revealing common procedural actions and key ingredients. Panel (g) shows the distribution of video durations, with most videos between 5-12 minutes, while panel (h) visualises a noun–verb co-occurrence heatmap, highlighting frequent action–ingredient pairings central to biryani preparation. Panels (i–l) depict canonical procedural steps identified via GPT-4-generated template recipes.
• Figure 3: Overview of the multimodal video segmentation and alignment pipeline. Panel (a) shows the 10-second clip segmentation of biryani cooking videos, where each segment is processed by InternVL-14B to extract visually grounded annotations of actions, ingredients, and utensils. Consecutive segments containing the same action are merged to form continuous spans, improving temporal coherence. Panel (b) presents example video scene graphs depicting detected entities and their relationships. displays an alignment heatmap between canonical recipe steps (vertical) and transcript sentences (horizontal), where colour intensity indicates semantic similarity and the red path represents the optimal sequence alignment computed via Dynamic Time Warping.
• Figure 4: Overview of the video comparison framework for biryani recipes. The framework operates through three sequential stages: Proposer (Qwen2-VL) generates plausible variations for each action, Frame Localiser (CLIP) identifies relevant frames, and Action Differencer compares frame pairs to detect differences. This example demonstrates analysis of "Adding ginger-garlic paste," identifying that Video B uses less paste than Video A.
• Figure 5: Visualisation of cooking process variations between Hyderabadi and Lucknowibiryani across several cooking stages. Each coloured section represents a major cooking stage, with individual squares showing specific actions. The opacity of the square is proportional to the degree of variation detected between the two biryani styles, where larger squares indicate more significant differences.