Toyteller: AI-powered Visual Storytelling Through Toy-Playing with Character Symbols

TL;DR

Toyteller introduces AI-powered visual storytelling driven by toy-playing with two symbolic characters. By learning a translational action space that links symbol motions and story text, it enables motion-to-text and text-to-motion generation, achieving real-time, interactive storytelling that outperforms GPT-4o on several metrics. A user study shows toy-playing expresses underdeveloped intentions and best complements natural language prompts, suggesting a hybrid, multimodal approach for creative tasks. The work also maps a design space for toy-playing interactions and discusses implications for human-AI interaction research, with practical pathways for extending spatial/temporal mappings, scene complexity, and form factors.

Abstract

We introduce Toyteller, an AI-powered storytelling system where users generate a mix of story text and visuals by directly manipulating character symbols like they are toy-playing. Anthropomorphized symbol motions can convey rich and nuanced social interactions; Toyteller leverages these motions (1) to let users steer story text generation and (2) as a visual output format that accompanies story text. We enabled motion-steered text generation and text-steered motion generation by mapping motions and text onto a shared semantic space so that large language models and motion generation models can use it as a translational layer. Technical evaluations showed that Toyteller outperforms a competitive baseline, GPT-4o. Our user study identified that toy-playing helps express intentions difficult to verbalize. However, only motions could not express all user intentions, suggesting combining it with other modalities like language. We discuss the design space of toy-playing interactions and implications for technical HCI research on human-AI interaction.

Figures (19)

• Figure 1: Heider and Simmel's experiment.
• Figure 2: The Toyteller interface consists of a timeline module (a), playground (b), and a button to enter the setting page (c). The user can use the progress bar (a-1) to navigate recorded motion frames, which align with story textboxes (a-2). The user can manipulate character symbols (b-1) to record character motions and eventually initiate text generation. The user can ask Toyteller to generate both character motion and story text (a-3) while giving a high-level direction as a natural language prompt (a-4). The user can also initiate the generation of story text only (a-5) and adjust the size of the last textbox (a-6). By default, Toyteller automatically reacts to the user's input, but the user can turn this off (a-7). The user can delete frames and textboxes after a selected frame with a button (a-8). After recording frames and text, the user can play them back (a-9).
• Figure 3: Interface for setting up the story scene. The paint and sparkle buttons allow users to use AI to generate setting-appropriate images and textual descriptions, respectively.
• Figure 4: The user can manually initiate motion-to-text generation with a) and b). b) allows users to generate a sentence by swapping the "active" character of the event recognized by the system.
• Figure 5: When text is provided before motions, with the highlighted button, the user can make AI generate motions.