Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

PodBench: A Comprehensive Benchmark for Instruction-Aware Audio-Oriented Podcast Script Generation

Chenning Xu, Mao Zheng, Mingyu Zheng, Mingyang Song

TL;DR

PodBench provides a specialized benchmark for instruction-aware, context-grounded podcast script generation with 800 long-context samples spanning 12 domains. It introduces a two-stage evaluation framework that separates instruction following from podcast-specific quality, combining deterministic checks with LLM-based judgments. Key findings show proprietary models often excel overall, while open-source models with explicit reasoning are more robust to long contexts and multi-speaker coordination, though content depth remains a bottleneck. The framework offers a reproducible testbed to study long-form, audio-centric generation and to guide future AI-podcast research.

Abstract

Podcast script generation requires LLMs to synthesize structured, context-grounded dialogue from diverse inputs, yet systematic evaluation resources for this task remain limited. To bridge this gap, we introduce PodBench, a benchmark comprising 800 samples with inputs up to 21K tokens and complex multi-speaker instructions. We propose a multifaceted evaluation framework that integrates quantitative constraints with LLM-based quality assessment. Extensive experiments reveal that while proprietary models generally excel, open-source models equipped with explicit reasoning demonstrate superior robustness in handling long contexts and multi-speaker coordination compared to standard baselines. However, our analysis uncovers a persistent divergence where high instruction following does not guarantee high content substance. PodBench offers a reproducible testbed to address these challenges in long-form, audio-centric generation.

PodBench: A Comprehensive Benchmark for Instruction-Aware Audio-Oriented Podcast Script Generation

TL;DR

PodBench provides a specialized benchmark for instruction-aware, context-grounded podcast script generation with 800 long-context samples spanning 12 domains. It introduces a two-stage evaluation framework that separates instruction following from podcast-specific quality, combining deterministic checks with LLM-based judgments. Key findings show proprietary models often excel overall, while open-source models with explicit reasoning are more robust to long contexts and multi-speaker coordination, though content depth remains a bottleneck. The framework offers a reproducible testbed to study long-form, audio-centric generation and to guide future AI-podcast research.

Abstract

Podcast script generation requires LLMs to synthesize structured, context-grounded dialogue from diverse inputs, yet systematic evaluation resources for this task remain limited. To bridge this gap, we introduce PodBench, a benchmark comprising 800 samples with inputs up to 21K tokens and complex multi-speaker instructions. We propose a multifaceted evaluation framework that integrates quantitative constraints with LLM-based quality assessment. Extensive experiments reveal that while proprietary models generally excel, open-source models equipped with explicit reasoning demonstrate superior robustness in handling long contexts and multi-speaker coordination compared to standard baselines. However, our analysis uncovers a persistent divergence where high instruction following does not guarantee high content substance. PodBench offers a reproducible testbed to address these challenges in long-form, audio-centric generation.
Paper Structure (38 sections, 5 figures, 7 tables)

This paper contains 38 sections, 5 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. AI-Podcast
  4. LLM-based Generative Writing
  5. PodBench
  6. Task Definition
  7. Benchmark Construction
  8. Input Material Collection.
  9. User Instruction Synthesis.
  10. Sample Filtering.
  11. Benchmark Analysis
  12. Evaluation Framework
  13. Stage 1: Instruction Following
  14. Stage 2: Podcast Script Quality
  15. Score Aggregation
  16. ...and 23 more sections

Figures (5)

  • Figure 1: The construction pipeline of PodBench.
  • Figure 2: Domain categories of input documents.
  • Figure 3: Overview of the evaluation framework.
  • Figure 4: The comparison of model performance on samples of different input length.
  • Figure 5: The comparison of model performance on samples of different speaker number. '1', '2' and '3-4' indicates the required speaker number in the user instructions.