Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

In Situ AI Prototyping: Infusing Multimodal Prompts into Mobile Settings with MobileMaker

Savvas Petridis, Michael Xieyang Liu, Alexander J. Fiannaca, Vivian Tsai, Michael Terry, Carrie J. Cai

TL;DR

The findings suggest that MobileMaker prototypes enabled more serendipitous discovery of: model input edge cases, discrepancies between AI’s and user’s in-context interpretation of the task, and contextual signals missed by the AI.

Abstract

Recent advances in multimodal large language models (LLMs) have made it easier to rapidly prototype AI-powered features, especially for mobile use cases. However, gathering early, mobile-situated user feedback on these AI prototypes remains challenging. The broad scope and flexibility of LLMs means that, for a given use-case-specific prototype, there is a crucial need to understand the wide range of in-the-wild input users are likely to provide and their in-context expectations for the AI's behavior. To explore the concept of in situ AI prototyping and testing, we created MobileMaker: a platform that enables designers to rapidly create and test mobile AI prototypes directly on devices. This tool also enables testers to make on-device, in-the-field revisions of prototypes using natural language. In an exploratory study with 16 participants, we explored how user feedback on prototypes created with MobileMaker compares to that of existing prototyping tools (e.g., Figma, prompt editors). Our findings suggest that MobileMaker prototypes enabled more serendipitous discovery of: model input edge cases, discrepancies between AI's and user's in-context interpretation of the task, and contextual signals missed by the AI. Furthermore, we learned that while the ability to make in-the-wild revisions led users to feel more fulfilled as active participants in the design process, it might also constrain their feedback to the subset of changes perceived as more actionable or implementable by the prototyping tool.

In Situ AI Prototyping: Infusing Multimodal Prompts into Mobile Settings with MobileMaker

TL;DR

The findings suggest that MobileMaker prototypes enabled more serendipitous discovery of: model input edge cases, discrepancies between AI’s and user’s in-context interpretation of the task, and contextual signals missed by the AI.

Abstract

Recent advances in multimodal large language models (LLMs) have made it easier to rapidly prototype AI-powered features, especially for mobile use cases. However, gathering early, mobile-situated user feedback on these AI prototypes remains challenging. The broad scope and flexibility of LLMs means that, for a given use-case-specific prototype, there is a crucial need to understand the wide range of in-the-wild input users are likely to provide and their in-context expectations for the AI's behavior. To explore the concept of in situ AI prototyping and testing, we created MobileMaker: a platform that enables designers to rapidly create and test mobile AI prototypes directly on devices. This tool also enables testers to make on-device, in-the-field revisions of prototypes using natural language. In an exploratory study with 16 participants, we explored how user feedback on prototypes created with MobileMaker compares to that of existing prototyping tools (e.g., Figma, prompt editors). Our findings suggest that MobileMaker prototypes enabled more serendipitous discovery of: model input edge cases, discrepancies between AI's and user's in-context interpretation of the task, and contextual signals missed by the AI. Furthermore, we learned that while the ability to make in-the-wild revisions led users to feel more fulfilled as active participants in the design process, it might also constrain their feedback to the subset of changes perceived as more actionable or implementable by the prototyping tool.
Paper Structure (38 sections, 11 figures, 1 table)

This paper contains 38 sections, 11 figures, 1 table.

Table of Contents

  1. Introduction
  2. Related Work
  3. Prototyping with AI
  4. In Situ User Testing
  5. Iterative Design Based on User Feedback
  6. Formative Studies & Design Goals
  7. MobileMaker
  8. Creating Prototypes
  9. Prototyping with Natural Language
  10. Creating with Natural Language
  11. Revising with Natural Language
  12. Prototype Testing Experience on Mobile
  13. Dynamic Prototype Revision
  14. Prototype Revision Dashboard
  15. Exploratory User Study
  16. ...and 23 more sections

Figures (11)

  • Figure 1: Testing AI prototypes on desktop, e.g. using UI mockups and LLM prompt editors (\ref{['traditional']}) vs. on mobile (\ref{['insitu']}): MobileMaker helps designers quickly get functional AI prototypes onto mobile devices and experienced in the wild for early feedback, and enables testers to revise and re-configure the AI prototype on-the-fly, while in the field.
  • Figure 2: MobileMaker UI. To build a prototype, users can add input (A), action (B), and output (C) widgets and customize their properties (D), e.g., editing an output widget's prompt (D1). Alternatively, users can create or revise prototypes with natural language via the "Revise with AI" panel (E). Changes are immediately rendered and can be tested in the mobile preview (F). Finally, users can test and revise the prototype on their phones in the wild (G).
  • Figure 3: Example prototype JSON representation.
  • Figure 4: Prototype Revision Dashboard
  • Figure 5: Questionnaire results comparing the two conditions. Bars are standard error and an asterisk indicates a statistically significant difference (after full Bonferroni correction).
  • ...and 6 more figures