VLM-SlideEval: Evaluating VLMs on Structured Comprehension and Perturbation Sensitivity in PPT

TL;DR

VLM-SlideEval tackles evaluating vision-language models on structured comprehension and perturbation sensitivity for PPT slides. It introduces a slide-native framework with ground-truth PPTX extraction, a unified schema, and three evaluation axes: pixel-level element extraction, robustness to controlled perturbations governed by severity $s\in[0,1]$, and deck-level narrative ordering. Empirically, five VLMs show limited pixel-accurate extraction and narrative coherence, though they perform better on single-slide content; perturbations reveal a fidelity-consistency trade-off across geometry, style, and especially text. The framework argues for calibrated, slide-native evaluators as critic-in-the-loop signals to guide iterative refinement in agentic pipelines.

Abstract

Vision-language models (VLMs) are increasingly used to evaluate multimodal content, including presentation slides, yet their slide-specific understanding remains underexplored {despite their growing role as critics in agentic, model-forward pipelines}. We introduce VLM-SlideEval, an evaluation framework that probes VLMs along three axes: (1) element-level extraction from slide images aligned to ground truth; (2) robustness to controlled perturbations in geometry, style, and text; and (3) higher-level comprehension, such as recovering a deck's narrative order from shuffled slides. Using publicly available decks from Zenodo (https://huggingface.co/datasets/Forceless/Zenodo10K/viewer/default/pptx), we standardize ground-truth element metadata from PowerPoint XML and live renderings into a unified, verifiable schema. Empirically, VLMs underperform on pixel-accurate extraction and show non-trivial agreement, fidelity, and consistency under controlled perturbations, while performing better on single-slide content understanding; however, they do not reliably capture narrative structure across slides. These results highlight the limits of current VLMs for slide evaluation and motivate calibrated, critic-in-the-loop evaluators that drive iterative refinement and selection in agentic pipelines.

Figures (8)

• Figure 1: Evaluation Task Examples: Top: From an original slide (A), we parse a simplified schema JSON (Table \ref{['tab:gt_schema']}) (B), reconstruct a normalized slide (C; blue dashed boxes show theme-embedded content omitted by the schema). A VLM predicts the schema from the re-rendered slide (D), and we score accuracy. Bottom: We subsample 100 decks, retain slides with $\geq 3$ visible elements (234 slides total), and apply perturbations to geometry, text, and style with severity $s\in[0,1]$ (larger $s$ means stronger changes; details in §\ref{['sec:method']}). Perturbed slides are then used for VLM quality evaluation and sensitivity analyses (§\ref{['sec:result']}).
• Figure 2: Parsed-only (solid) vs. e2e (hatched) with coverage (i.e., fraction of ground truth instances evaluated for the metric; lines). o3/gpt-5 lead on Matching F1 (0.71-0.72) and Text Content F1 (0.76-0.78); o3 best in geometry (1-IoU $0.55$). Font Family Accuracy is low overall (max $0.42$). More results in Fig. \ref{['fig:other_bars']}, Appendix \ref{['app:detailed_results']}.
• Figure 3: Evaluation results of model behavior under controlled perturbations.
• Figure 4: Prompt used for structured extractions from VLMs for a single slide image.
• Figure 5: Prompt template used by VLM evaluators on perturbed slides.