Uncertainty-boosted Robust Video Activity Anticipation

TL;DR

The paper tackles data uncertainty in video activity anticipation by introducing an uncertainty-boosted framework that outputs a credibility score for predictions. It uses a dual-branch design to compute target distributions and a model-derived uncertainty, applying temperature scaling and a co-occurrence–aware target label space informed by temporal correlations and ConceptNet relations. Relative uncertainty learning across samples and time, plus distribution adjustment and specialized losses, yield improved robustness and interpretability across multiple datasets and backbones, particularly on uncertain and long-tail classes. The approach is plug-and-play, scalable to existing generative methods, and offers practical implications for trustworthy long-horizon video understanding with broader applicability to video tasks.

Abstract

Video activity anticipation aims to predict what will happen in the future, embracing a broad application prospect ranging from robot vision and autonomous driving. Despite the recent progress, the data uncertainty issue, reflected as the content evolution process and dynamic correlation in event labels, has been somehow ignored. This reduces the model generalization ability and deep understanding on video content, leading to serious error accumulation and degraded performance. In this paper, we address the uncertainty learning problem and propose an uncertainty-boosted robust video activity anticipation framework, which generates uncertainty values to indicate the credibility of the anticipation results. The uncertainty value is used to derive a temperature parameter in the softmax function to modulate the predicted target activity distribution. To guarantee the distribution adjustment, we construct a reasonable target activity label representation by incorporating the activity evolution from the temporal class correlation and the semantic relationship. Moreover, we quantify the uncertainty into relative values by comparing the uncertainty among sample pairs and their temporal-lengths. This relative strategy provides a more accessible way in uncertainty modeling than quantifying the absolute uncertainty values on the whole dataset. Experiments on multiple backbones and benchmarks show our framework achieves promising performance and better robustness/interpretability. Source codes are available at https://github.com/qzhb/UbRV2A.

Figures (12)

• Figure 1: The activity video evolution uncertainty phenomenon on EPIC-KITCHENS-55 dataset. For clarity, partial classes are shown.
• Figure 2: The uncertainty-boosted activity anticipation framework. We use $F_c$ and $F_u$ to produce the probability distribution of the anticipation result and the uncertainty vector. Then we use $\hat{u}^i_t$, the mean of the uncertainty vector, to adjust the smoothness of the distribution ($S$) and obtain $\hat{p}^i_t$.
• Figure 3: The setting of the video activity anticipation task.
• Figure 4: Given a probability distribution of the anticipation result, we visualize the effects of the distribution adjustment strategy.
• Figure 5: The sample-wise relative uncertainty learning pipeline. We obtain $\hat{f}^{ij}_t$ by mixing the anticipated features with their relative uncertainty values.