Vistoria: A Multimodal System to Support Fictional Story Writing through Instrumental Text-Image Co-Editing

TL;DR

The paper tackles the limitation of text-centric fiction-writing tools by introducing Vistoria, a multimodal system for synchronized image–text co-editing grounded in Instrumental Interaction and Dual Coding Theory. Through a formative Wizard-of-Oz study with 10 writers and a controlled usability study with 12 participants, the authors demonstrate that multimodal co-editing enhances expressiveness, immersion, and divergent ideation, while increasing cognitive workload. Vistoria uses instrument-based operations (Lasso, Collage, Perspective Shift, Filter) to create image–text cards and a persistent cluster-based workspace, enabling cross-modal alignment and iterated narrative development. The findings suggest significant potential for dynamic, mixed-initiative writing tools that balance abstraction and concreteness, providing tangible guidance for future design of multimodal creativity support systems.

Abstract

Humans think visually-we remember in images, dream in pictures, and use visual metaphors to communicate. Yet, most creative writing tools remain text-centric, limiting how authors plan and translate ideas. We present Vistoria, a system for synchronized text-image co-editing in fictional story writing that treats visuals and text as coequal narrative materials. A formative Wizard-of-Oz co-design study with 10 story writers revealed how sketches, images, and annotations serve as essential instruments for ideation and organization. Drawing on theories of Instrumental Interaction and Structural Mapping, Vistoria introduces multimodal operations-lasso, collage, filters, and perspective shifts that enable seamless narrative exploration across modalities. A controlled study with 12 participants shows that co-editing enhances expressiveness, immersion, and collaboration, enabling writers to explore divergent directions, embrace serendipitous randomness, and trace evolving storylines. While multimodality increased cognitive demand, participants reported stronger senses of authorship and agency. These findings demonstrate how multimodal co-editing expands creative potential by balancing abstraction and concreteness in narrative development.

Figures (13)

• Figure 1: WoZ System and Study Process (the example of P4) (a) The user edits text in the Text editor; (b) The user writes, sketches, and speaks out their intents; (c)-(d) both wizards paste in the user's inputs into AI software windows; (e) wizards pasted the generated results back onto the canvas.
• Figure 2: Vistoria’s interface: a left Text Editor, a central Cluster panel (can be collapsed when not used), and a right free-form Canvas.
• Figure 3: Vistoria analyzes selected text, sketches, images, and broader story context to generate image–text cards that externalize narrative development directions.
• Figure 4: A set of instrumental operations for image-text co-editing to enhance planning and translating of fictional story writing: (a) Lasso selects regions for coupled image–text edits. (b) Collage enables writers to extract elements and compose across cards to discover new narrative directions. (c) Perspective Shift changes an image’s viewpoint and automatically regenerates the story’s point of view (first/third/second person). (d) Filters align visual style and textual tone (e.g., melancholic/dreamy) by jointly altering image effects and rewriting prose.
• Figure 5: Writers highlight objects and text segments on cards (b-1); the Cluster panel aggregates these by character/object/scene and can auto-summarize settings/plot/description about a certain object to guide final writing (b-2); Clicking on a specific image reveals the corresponding highlights and comments from earlier phases left on Canvas (b-3).