Gensors: Authoring Personalized Visual Sensors with Multimodal Foundation Models and Reasoning

TL;DR

The paper addresses the bottleneck of enabling everyday users to define personalized, real-time AI sensors for visual streams by moving beyond prompting to a criteria-based reasoning framework. It introduces Gensors, a system that decomposes high-level sensing tasks into independent criteria, automates initial criterion generation, supports example-driven specification, and provides test-case scaffolding, all powered by multimodal foundation models in a two-stage Gemini pipeline. A formative study and a formal user study demonstrate that this approach yields greater user sense of control, understanding, and ability to debug compared with prompting alone, while revealing challenges such as hallucinations and response flickering inherent to MLLMs. The work contributes design goals, a practical UI/UX for criterion-level authoring, and empirical evidence that structured, criterion-based reasoning can improve personal AI sensor definition, with implications for intuitive and customizable sensing in home environments. The findings highlight both the potential and the limitations of MLLMs for user-driven sensing and outline directions for robust, privacy-conscious, on-device or tightly controlled cloud-based deployments and richer logical constructs.

Abstract

Multimodal large language models (MLLMs), with their expansive world knowledge and reasoning capabilities, present a unique opportunity for end-users to create personalized AI sensors capable of reasoning about complex situations. A user could describe a desired sensing task in natural language (e.g., "alert if my toddler is getting into mischief"), with the MLLM analyzing the camera feed and responding within seconds. In a formative study, we found that users saw substantial value in defining their own sensors, yet struggled to articulate their unique personal requirements and debug the sensors through prompting alone. To address these challenges, we developed Gensors, a system that empowers users to define customized sensors supported by the reasoning capabilities of MLLMs. Gensors 1) assists users in eliciting requirements through both automatically-generated and manually created sensor criteria, 2) facilitates debugging by allowing users to isolate and test individual criteria in parallel, 3) suggests additional criteria based on user-provided images, and 4) proposes test cases to help users "stress test" sensors on potentially unforeseen scenarios. In a user study, participants reported significantly greater sense of control, understanding, and ease of communication when defining sensors using Gensors. Beyond addressing model limitations, Gensors supported users in debugging, eliciting requirements, and expressing unique personal requirements to the sensor through criteria-based reasoning; it also helped uncover users' "blind spots" by exposing overlooked criteria and revealing unanticipated failure modes. Finally, we discuss how unique characteristics of MLLMs--such as hallucinations and inconsistent responses--can impact the sensor-creation process. These findings contribute to the design of future intelligent sensing systems that are intuitive and customizable by everyday users.

Figures (5)

• Figure 1: Gensors' main user interface for formulating and curating criteria that govern a sensor's behavior. For a high-level sensing task (in this case "tell me if my toddler might damage something"), the system operates as follows: At each time interval, the sensor evaluates all criteria (c) simultaneously using frames captured by a live camera (a) over the past few seconds (b). The result for each criterion is displayed as a green or red chip, indicating a positive or negative outcome, respectively. Users can click on a chip to view detailed results for that specific criterion, which is the description from the MLLM's interpretation of the scene (d). The sensor then synthesizes these individual results to make an informed final decision regarding the original sensing task (e). In the criteria editor (f), users have the option to either add their own criterion (f1) or have Gensors automatically generate criteria (f2) based on the initial sensing task and the live camera view. To modify an existing criterion (i), users can update its description (i1), and Gensors will automatically generate a title (i2) for display in the Live sensor view (c). Furthermore, users can 1) add additional text (h1) or visual examples (h2) to explain a criterion (h) based on their personal context; 2) review Gensors-generated suggestions for testing a specific criterion (g); 3) temporarily enable or disable a criterion (j); and 4) configure how the sensor reaches its final verdict (k), ranging from allowing the MLLM to make an intelligent decision based on all criteria results (k1), to rule-based combinations using boolean logic (k2, k3).
• Figure 2: The prompt-editor authoring tool participants used in the formative study, as well as the baseline condition in the user study (see Section \ref{['user-study-procedure']}). Users can view what their camera currently sees (a) and pause and play the sensor (b). They can edit their prompt (d) and then view its latest result (e) as well as the corresponding input image it was run on (c).
• Figure 3: The Examples-diff feature in Gensors allows users to provide visual examples for each answer category (a and b) related to the high-level sensing task. Gensors then presents its reasoning process (c) and any additional criteria generated from that reasoning (d). Users can choose whether to incorporate these criteria into the main list.
• Figure 4: Comparing Gensors against baseline. Gensors had significantly higher ratings for Control, Understand Capabilities, and Communicate Requirements. (Bars are standard error and * indicates statistically significant difference, after full Bonferroni correction).
• Figure 5: Participants' feedback on Gensors features. Of Gensors' features, all were considered helpful, with the automatically generated criteria perceived to be the most helpful.