Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

SPoRC-VIST: A Benchmark for Evaluating Generative Natural Narrative in Vision-Language Models

Yunlin Zeng

TL;DR

This work introduces SPoRC-VIST, a benchmark that trains on synthetic imagery derived from high-quality podcast transcripts (SPoRC) and evaluates on real VIST image sequences to study visual narrative generation. By fine-tuning a relatively compact Qwen3-VL-32B model with LoRA and NEFTune, the authors achieve richer, more natural podcast-style dialogues that maintain visual grounding, outperforming a much larger base model in conversational depth and perceived naturalness. The paper also proposes novel style metrics (average turn length, speaker switch rate) and an AI-as-a-Judge evaluation paradigm, demonstrating robust generalization to real images and offering a more task-relevant assessment than traditional n-gram metrics. These contributions advance open-ended multimodal storytelling and provide practical benchmarks and evaluation methods for future VLMs in creative, narrative-driven tasks.

Abstract

Vision-Language Models (VLMs) have achieved remarkable success in descriptive tasks such as image captioning and visual question answering (VQA). However, their ability to generate engaging, long-form narratives -- specifically multi-speaker podcast dialogues -- remains under-explored and difficult to evaluate. Standard metrics like BLEU and ROUGE fail to capture the nuances of conversational naturalness, personality, and narrative flow, often rewarding safe, repetitive outputs over engaging storytelling. In this work, we present a novel pipeline for end-to-end visual podcast generation, and fine-tune a Qwen3-VL-32B model on a curated dataset of 4,000 image-dialogue pairs. Crucially, we use a synthetic-to-real training strategy: we train on high-quality podcast dialogues from the Structured Podcast Research Corpus (SPoRC) paired with synthetically generated imagery, and evaluate on real-world photo sequences from the Visual Storytelling Dataset (VIST). This rigorous setup tests the model's ability to generalize from synthetic training data to real-world visual domains. We propose a comprehensive evaluation framework that moves beyond textual overlap, and use AI-as-a-judge (Gemini 3 Pro, Claude Opus 4.5, GPT 5.2) and novel style metrics (average turn length, speaker switch rate) to assess quality. Our experiments demonstrate that our fine-tuned 32B model significantly outperforms a 235B base model in conversational naturalness ($>$80\% win rate) and narrative depth (+50\% turn length), while maintaining identical visual grounding capabilities (CLIPScore: 20.39).

SPoRC-VIST: A Benchmark for Evaluating Generative Natural Narrative in Vision-Language Models

TL;DR

This work introduces SPoRC-VIST, a benchmark that trains on synthetic imagery derived from high-quality podcast transcripts (SPoRC) and evaluates on real VIST image sequences to study visual narrative generation. By fine-tuning a relatively compact Qwen3-VL-32B model with LoRA and NEFTune, the authors achieve richer, more natural podcast-style dialogues that maintain visual grounding, outperforming a much larger base model in conversational depth and perceived naturalness. The paper also proposes novel style metrics (average turn length, speaker switch rate) and an AI-as-a-Judge evaluation paradigm, demonstrating robust generalization to real images and offering a more task-relevant assessment than traditional n-gram metrics. These contributions advance open-ended multimodal storytelling and provide practical benchmarks and evaluation methods for future VLMs in creative, narrative-driven tasks.

Abstract

Vision-Language Models (VLMs) have achieved remarkable success in descriptive tasks such as image captioning and visual question answering (VQA). However, their ability to generate engaging, long-form narratives -- specifically multi-speaker podcast dialogues -- remains under-explored and difficult to evaluate. Standard metrics like BLEU and ROUGE fail to capture the nuances of conversational naturalness, personality, and narrative flow, often rewarding safe, repetitive outputs over engaging storytelling. In this work, we present a novel pipeline for end-to-end visual podcast generation, and fine-tune a Qwen3-VL-32B model on a curated dataset of 4,000 image-dialogue pairs. Crucially, we use a synthetic-to-real training strategy: we train on high-quality podcast dialogues from the Structured Podcast Research Corpus (SPoRC) paired with synthetically generated imagery, and evaluate on real-world photo sequences from the Visual Storytelling Dataset (VIST). This rigorous setup tests the model's ability to generalize from synthetic training data to real-world visual domains. We propose a comprehensive evaluation framework that moves beyond textual overlap, and use AI-as-a-judge (Gemini 3 Pro, Claude Opus 4.5, GPT 5.2) and novel style metrics (average turn length, speaker switch rate) to assess quality. Our experiments demonstrate that our fine-tuned 32B model significantly outperforms a 235B base model in conversational naturalness (80\% win rate) and narrative depth (+50\% turn length), while maintaining identical visual grounding capabilities (CLIPScore: 20.39).
Paper Structure (30 sections, 4 figures, 6 tables)

This paper contains 30 sections, 4 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Visual Storytelling
  4. Vision-Language Models
  5. Generative Evaluation
  6. The SPoRC-VIST Benchmark
  7. Data Sources
  8. Why Real Transcripts? A Preliminary Study
  9. Dataset Construction Pipeline
  10. Methodology
  11. Model Architecture
  12. Fine-Tuning Strategy
  13. Experiments
  14. Implementation Details
  15. Metrics
  16. ...and 15 more sections

Figures (4)

  • Figure 1: Two evaluation sequences from the VIST dataset. Each row shows five real-world photos with their original single-sentence VIST captions. Row 1 depicts a wedding celebration; Row 2 shows motorcycle riding and marathon running. Our goal is to transform these brief, factual captions into rich, multi-turn podcast dialogues. Note: The model is trained on synthetic images (Stable Diffusion) but evaluated on these real photos to test domain generalization.
  • Figure 2: Images from our preliminary study. We generated these images from a real podcast excerpt about exploring an abandoned house. We then asked AI models to generate transcripts from the images. The original human transcript consistently outperformed AI-generated alternatives, motivating our use of real transcripts as training targets.
  • Figure 3: Dataset construction pipeline. Left: A podcast excerpt (555 words) with rich visual descriptions is extracted from SPoRC. Middle: Claude Sonnet 4.5 generates five scene-specific image prompts. Right: Stable Diffusion 3.5 renders the prompts into coherent visual sequences. This process creates 4,000 image-dialogue training pairs.
  • Figure 4: Qualitative comparison of generated transcripts on the same wedding photo sequence (\ref{['fig:teaser']}). 32B Base: Conversational but observational, directly describing photos ("look at this first one," "that second photo"). 32B Fine-tuned: Most natural style---speakers riff on meta-topics, share personal anecdotes (the sandwich guy, the drunk best man), and exhibit genuine humor without referencing images. 235B Base: Engaging but overly descriptive---phrases like "the wood paneling, the chandeliers, the judge's bench" read more like captions than natural speech.