Facilitating Video Story Interaction with Multi-Agent Collaborative System

TL;DR

This work tackles the challenge of delivering deep, personalized interactive experiences in video storytelling by integrating a Vision-Language understanding pipeline with Retrieval-Augmented Generation and a Multi-Agent System to support evolving characters and customizable scenes. The authors ground the design in a formative study, then implement a cross-modal processing framework that informs a growth-aware MAS, enabling trans-temporal chats and user-driven scene visualization, demonstrated through a Harry Potter case study. Key contributions include a VLM-based cross-modal understanding approach, a RAG+MAS interaction architecture, and an interactive interface supporting stage- and time-aware storytelling with emergent character growth. The approach advances personalized, multi-character narrative exploration and could transform serialized and biographical video content by enabling dynamic, user-tailored experiences.

Abstract

Video story interaction enables viewers to engage with and explore narrative content for personalized experiences. However, existing methods are limited to user selection, specially designed narratives, and lack customization. To address this, we propose an interactive system based on user intent. Our system uses a Vision Language Model (VLM) to enable machines to understand video stories, combining Retrieval-Augmented Generation (RAG) and a Multi-Agent System (MAS) to create evolving characters and scene experiences. It includes three stages: 1) Video story processing, utilizing VLM and prior knowledge to simulate human understanding of stories across three modalities. 2) Multi-space chat, creating growth-oriented characters through MAS interactions based on user queries and story stages. 3) Scene customization, expanding and visualizing various story scenes mentioned in dialogue. Applied to the Harry Potter series, our study shows the system effectively portrays emergent character social behavior and growth, enhancing the interactive experience in the video story world.

Figures (15)

• Figure 1: Design framework for video story interaction using MAS. Based on a semi-structured interview in the formative study and the six elements theory of complete narrative.
• Figure 2: System overview. The system involves a machine understanding a video story and incorporating that understanding into Human-Computer Interaction (HCI) using Multi-Agent Systems (MAS). Users can select any stage of the video story to interact with. This interaction includes chatting with characters at that stage and customizing potential scenes mentioned in the conversation.
• Figure 3: VLM-based video story comprehension pipeline. The use of a Vision-Language Model (VLM) allows machines to comprehend video stories like humans. The process begins with the original video input, integrates visual and plot details, and organizes the information into three modalities. The understood information is then sequentially stored in stage-specific text files.
• Figure 4: The input and output utilize MAS as the main component. Once the machine comprehends the video story, MAS handles and executes all the interactive experiences. We employ three MAS communication structures based on specific tasks: shared message pool, decentralized, and layered.
• Figure 5: Multi space chat design concept. The integration of fluid narrative and high-freedom interactive spaces using the startCoroutine( ) design. The MAS updates stored information alongside the narrative, ensuring that users stay aware of the story's progress and preventing fragmentation between interaction and narrative. Users can choose any stage, enter the interaction space, watch a short video, and interact with characters at different growth stages.