SPICE: Smart Projection Interface for Cooking Enhancement

TL;DR

SPICE presents a tangible, projection-based interface for cooking that integrates an optical tracking system, an agent-based visualization platform, and vision–language models to project recipe guidance directly onto a cooking surface. The system architecture spans sensing ($PC_{1}$), computation ($PC_{2}$, $ROS$) and actuation (short-throw projector), with ingredients and instruments represented as interactive agents. In a 30-participant study, SPICE reduced task duration and the number of stops while improving self-reported Taste, Confidence, and Efficiency, though first-time users sometimes perceived lower efficiency due to unfamiliarity; results support the potential of TUIs to enhance everyday, two-handed tasks. The work highlights a path toward seamless physical–digital blending in kitchen environments and suggests broader applications in other high-stakes, hands-on domains, while acknowledging cost and usability considerations for wider adoption.

Abstract

Tangible User Interfaces (TUI) for human--computer interaction (HCI) provide the user with physical representations of digital information with the aim to overcome the limitations of screen-based interfaces. Although many compelling demonstrations of TUIs exist in the literature, there is a lack of research on TUIs intended for daily two-handed tasks and processes, such as cooking. In response to this gap, we propose SPICE (Smart Projection Interface for Cooking Enhancement). SPICE investigates TUIs in a kitchen setting, aiming to transform the recipe following experience from simply text-based to tangibly interactive. SPICE uses a tracking system, an agent-based simulation software, and vision large language models to create and interpret a kitchen environment where recipe information is projected directly onto the cooking surface. We conducted comparative usability and a validation studies of SPICE, with 30 participants. The results show that participants using SPICE completed the recipe with far less stops and in a substantially shorter time. Despite this, participants self-reported negligible change in feelings of difficulty, which is a direction for future research. Overall, the SPICE project demonstrates the potential of using TUIs to improve everyday activities, paving the way for future research in HCI and new computing interfaces.

Figures (5)

• Figure 1: System overview where its three main components are depicted. A) The sensing part of the system includes the use of an optical tracking system (i.e., OptiTrack) to track rigid bodies, a Rigid Body Interface (RBI) (i.e., a 3D-printed base that holds IR reflective markers), and a USB video camera to record the scene where experiments take place. B) The computational element is composed of three PCs. $PC_1$ is running the tracking system processing software (Motive3D), which sends information to $PC_2$ running ROS (Robot Operating System). Finally, $PC_3$ runs GAMA (an agent-based visualization software) that obtains information from $PC_2$ to compute and render the SPICE interface. C) The actuation component includes a short throw projector angled at the working space where the user can interact with the system by moving the RBI.
• Figure 2: Sensing elements. In A), we can see a representation of the experimental area used in this research. (i) shows the cooking table that acts as the cooking surface. (ii) a USB camera mounted above the table. (iii) An Optitrack system mounted on a truss angled towards the table. (iv) shows a Rigid Body Interface (RBI); a 3D printed piece that holds IR reflective markers detected by (iii). B) shows the CAD model with measures of the RBI. C) depicts one of the RBI after being 3D printed and attaching the retro-reflective markers.
• Figure 3: SPICE User Interface. This figure shows the layout of the User Interface (UI) used in the project. From left to right: (i) ingredients detected by the VLM are placed in separated boxes. (ii) The recipe title is displayed together with its corresponding steps (iii). Detailed instructions (iv) such as cooking times and ingredient weights are displayed below the corresponding step. Finally, the RBI (v) is placed at the bottom right of the cooking surface to scroll among the recipe steps.
• Figure 4: Overview of the experimental design. A first group of 20 participants (Experiment group) were invited first to follow the recipe on the smartphone (orange) and after a two-week interval follow the recipe by using SPICE (green). A second group of 10 participants (Validation group) exclusively used SPICE to follow the recipe (purple).
• Figure 5: Overview of the results obtained grouped by each one of the metrics considered. A) shows a boxplot with mean and standard deviations of the difficulty perceived for the experiment group that followed the recipe in the smartphone (orange), using SPICE (green), and the validation group (purple). B, C, D, and E show the same information for the Confidence, Taste, Efficiency, and Total Duration (in secs) metrics respectively.