Jigsaw: Authoring Immersive Storytelling Experiences with Augmented Reality and Internet of Things

TL;DR

Jigsaw addresses the challenge of making immersive storytelling accessible by integrating mobile AR with off-the-shelf IoT devices. It presents an authoring tool with a scene-based, open-ended trigger framework that lets novices both experience and craft narratives. The authors prototype three immersive stories (Benjamin Franklin Kite Experiment, The Wind and The Sun, Goodnight Moon) and conduct a qualitative study with 20 participants, showing that IoT augmentation enhances immersion and that playback is engaging, though sensory overload remains a key design concern. The work contributes an AR+IoT authoring tool, illustrative stories, and design guidance for future immersive storytelling systems.

Abstract

Augmented Reality (AR) presents new opportunities for immersive storytelling. However, this immersiveness faces two main hurdles. First, AR's immersive quality is often confined to visual elements, such as pixels on a screen. Second, crafting immersive narratives is complex and generally beyond the reach of amateurs due to the need for advanced technical skills. We introduce Jigsaw, a system that empowers beginners to both experience and craft immersive stories, blending virtual and physical elements. Jigsaw uniquely combines mobile AR with readily available Internet-of-things (IoT) devices. We conducted a qualitative study with 20 participants to assess Jigsaw's effectiveness in both consuming and creating immersive narratives. The results were promising: participants not only successfully created their own immersive stories but also found the playback of three such stories deeply engaging. However, sensory overload emerged as a significant challenge in these experiences. We discuss design trade-offs and considerations for future endeavors in immersive storytelling involving AR and IoT.

Figures (3)

• Figure 1: Three immersive stories designed via a mix of mobile AR and IoT devices. (a) The Benjamin Franklin story leverages IoT as background augmentation and enables users to actively participate as different roles in the story; (b) The Wind and the Sun story uses IoT as foreground augmentation and pictures users as the audience who watch the story proceeds; (c) The Goodnight Moon story uses IoT as foreground elements in the story, each of which responds to users' greeting.
• Figure 2: A step-by-step demonstration of creating the first three scenes of the Goodnight Moon story using Jigsaw's authoring tool.
• Figure 3: Authoring interfaces for adding narration, editing smart fans, and defining touch interactions. (a) For self-narrated stories like the Wind and the Sun, the default trigger for each scene is tapping on the screen and users can add narration to each scene. (b) Users can edit the intensity of the smart fan with the Wind character anchored to it in The Wind and the Sun story. (c) When defining the touch interaction in the Benjamin Franklin story, users can select which object to touch by tapping on the object. (d) Users can also choose built-in effects of 3D models when the touch interaction is triggered.