SimTube: Generating Simulated Video Comments through Multimodal AI and User Personas

TL;DR

SimTube is introduced, a generative AI system designed to simulate audience feedback in the form of video comments before a video's release that shows that SimTube's generated comments are not only relevant, believable, and diverse but often more detailed and informative than actual audience comments.

Abstract

Audience feedback is crucial for refining video content, yet it typically comes after publication, limiting creators' ability to make timely adjustments. To bridge this gap, we introduce SimTube, a generative AI system designed to simulate audience feedback in the form of video comments before a video's release. SimTube features a computational pipeline that integrates multimodal data from the video-such as visuals, audio, and metadata-with user personas derived from a broad and diverse corpus of audience demographics, generating varied and contextually relevant feedback. Furthermore, the system's UI allows creators to explore and customize the simulated comments. Through a comprehensive evaluation-comprising quantitative analysis, crowd-sourced assessments, and qualitative user studies-we show that SimTube's generated comments are not only relevant, believable, and diverse but often more detailed and informative than actual audience comments, highlighting its potential to help creators refine their content before release.

Figures (7)

• Figure 1: Users upload a video via the Upload Video Page, and (A) view the comments generated by SimTube on the Simulated Comment Page, and by (B) replying to a comment or (C) specifying a customized persona, users can ask SimTube to generate new comments. In addition, users can refer to the persona description by (D) hovering on the commenter's icon.
• Figure 2: Thread Expansion: Upon receipt of (A) the user's reply, The thread is expanded by both (B) the user's reply and (C) the generated response of the commenter. Persona Crafting: (D) Upon user specification of a persona, (E) a new comment is generated in alignment with the video content and the user-defined persona.
• Figure 3: SimTube leverage VLM and Whisper to process videos in visual and audio components per modality. Personas are then queried based on the video content for comment generation. The system diagrams, situated centrally in the figure, along with the intermediate outputs on both flanks, exemplify this process.
• Figure 4: This plot includes quantitative results from Crowd-Sourced Study and other automatic metrics measuring diversity such as BERTScore, Distinct N-grams, and Self-BLEU.
• Figure 5: This figure contains quantitative results from automatic metrics measuring relevance between targeted text and video content, such as LLM Eval, ROUGE-n (n=1, 2, L), and BERTScore.