Molecumentary: Scalable Narrated Documentaries Using Molecular Visualization

TL;DR

The paper addresses the challenge of making complex molecular visualizations comprehensible to non-experts while enabling scalable, future-proof science communication. It introduces the Scalable Documentary framework, combining real-time 3D visualization, automatic camera control, and on-demand synthetic narration to generate molecumentaries from heterogeneous molecular data. Core contributions include Story-Graph Foraging for automatic story-graph construction, and Real-Time Narrative Synthesis to produce adaptive narratives via two modes: self-guided traversal and text-to-molecumentary, demonstrated on meso-scale datasets such as HIV, Mycoplasma, and SARS-CoV-2. The results indicate feasible real-time performance and positive expert feedback, highlighting the approach’s potential to update and broaden science outreach resources through on-demand, narrated visuals integrated with domain knowledge.

Abstract

We present a method for producing documentary-style content using real-time scientific visualization. We produce molecumentaries, i.e., molecular documentaries featuring structural models from molecular biology. We employ scalable methods instead of the rigid traditional production pipeline. Our method is motivated by the rapid evolution of interactive scientific visualization, which shows great potential in science dissemination. Without some form of explanation or guidance, however, novices and lay-persons often find it difficult to gain insights from the visualization itself. We integrate such knowledge using the verbal channel and provide it along an engaging visual presentation. To realize the synthesis of a molecumentary, we provide technical solutions along two major production steps: 1) preparing a story structure and 2) turning the story into a concrete narrative. In the first step, information about the model from heterogeneous sources is compiled into a story graph. Local knowledge is combined with remote sources to complete the story graph and enrich the final result. In the second step, a narrative, i.e., story elements presented in sequence, is synthesized using the story graph. We present a method for traversing the story graph and generating a virtual tour, using automated camera and visualization transitions. Texts written by domain experts are turned into verbal representations using text-to-speech functionality and provided as a commentary. Using the described framework we synthesize automatic fly-throughs with descriptions that mimic a manually authored documentary. Furthermore, we demonstrate a second scenario: guiding the documentary narrative by a textual input.

Figures (10)

• Figure 1: In the tour of the HIV in blood plasma model we for example visit the capsid \ref{['fig:results-hiv2']} which contains the genetic information of the virus. Besides the RNA, the capsid contains several important proteins, such as Reverse Transcriptase \ref{['fig:results-hiv3']}.
• Figure 2: The Scalable Documentary concept: We provide visuals as a real-time visualization and couple it with an automatic camera traversing the 3D scene. We augment the fly-through with a synthetic commentary that we generate on-demand, as opposed to using a pre-recorded voice-over.
• Figure 3: Overview of the framework for generating molecumentaries. In describing the technical contributions we follow the distinction between a story (static representation of story elements) and a narrative (dynamic arrangement of these story elements). Story-graph foraging provides a scalable approach for compiling information about the model which can be used for storytelling, while the real-time narrative synthesis encompasses solutions for turning the story graph into a specific narrative at runtime.
• Figure 4: The three steps in story graph foraging. First, we build the structural skeleton. Second, we associate textual descriptions with individual nodes. Finally, we add functional relationships.
• Figure 5: A sample textual description in which a functional relationship has been extracted. Through keyword detection the fact that the capsid protein forms a structure protecting the RNA is established. Such a functional relationship is added to the story graph as an edge.