Explicit Abstention Knobs for Predictable Reliability in Video Question Answering

TL;DR

This work investigates confidence-based abstention for VideoQA, examining whether model confidence provides reliable control over error rates and remains robust under evidence degradation. Using NExT-QA with Gemini 2.0 Flash, the study shows that sweeping the abstention threshold $\varepsilon$ yields smooth, monotone risk-coverage curves and strong calibration within the baseline distribution (for example, $63.7\%$ coverage with $9.4\%$ error at $\varepsilon\approx0.71$ and $ECE\approx0.018$). However, when video evidence is degraded (18 frames down to 6), confidence does not contract as evidence quality diminishes, leading to a notable drop in coverage with only modest changes in conditional risk, indicating the gate is not epistemic. A logprob-based confidence analysis reveals that the token-level decision distribution over answer options is even less sensitive to evidence loss, suggesting representational limitations rather than interface artifacts. The authors argue for warrant-based selective prediction, where confidence is bounded by evidence-supported limits, and discuss deployment implications, threshold transfer across regimes, and avenues for fine-tuning to incorporate observability signals. Overall, the paper highlights the need for evidence-aware confidence mechanisms to achieve reliable and robust selective prediction in vision-language systems.

Abstract

High-stakes deployment of vision-language models (VLMs) requires selective prediction, where systems abstain when uncertain rather than risk costly errors. We investigate whether confidence-based abstention provides reliable control over error rates in video question answering, and whether that control remains robust under distribution shift. Using NExT-QA and Gemini 2.0 Flash, we establish two findings. First, confidence thresholding provides mechanistic control in-distribution. Sweeping threshold epsilon produces smooth risk-coverage tradeoffs, reducing error rates f

Figures (7)

• Figure 1: Example evidence packet showing 6 of 18 frames extracted from a 76.5-second video. Frames are sampled to provide both broad temporal coverage (uniform sampling) and focus on the middle third (zoom region). Timestamps are shown above each frame.
• Figure 2: Logprob-to-probability computation. Raw log probabilities $\ell_i$ from the model's token distribution are normalized via log-softmax over the five answer options. This renormalization is necessary because the model's full vocabulary distribution includes tokens beyond A--E; we restrict to the answer space and compute a proper probability distribution over only the valid options. The resulting $p_i$ values sum to 1 and represent the model's relative preference among answer choices.
• Figure 3: Visual comparison of evidence packets for the same video. Top row shows 6 frames sampled from the original 18-frame evidence packet. Bottom row shows all 6 frames from the degraded condition (Shift A). The degraded condition provides much sparser temporal coverage of the video's 76.5-second duration.
• Figure 4: Baseline performance. (a) Risk-Coverage curve showing smooth tradeoff with knee at 60--70% coverage; starred point marks $\varepsilon=0.71$ (9.4% risk, 63.7% coverage). (b) Reliability diagram at $\varepsilon=0.71$ showing good calibration in the high-confidence bin. (c) ECE decreases as threshold tightens, dropping from 0.067 to below 0.02.
• Figure 5: Concrete example of overconfidence under evidence degradation. The question requires observing sustained behavior across time. With 18 frames (6 shown here), the model correctly identifies the behavior (B) with confidence 1.00. With only 6 frames from the degraded condition, the model misses critical moments, answers incorrectly (A), yet reports confidence 0.70. The model fails to recognize that sparse sampling provides insufficient evidence.