Video Set Distillation: Information Diversification and Temporal Densification

TL;DR

This work tackles the dual redundancy challenge in video sets by introducing Video Set Distillation and the IDTD framework. IDTD jointly performs information diversification (via a shared Feature Pool and multiple Feature Selectors) and temporal densification (via a Temporal Fusor with Stochastic Temporal Augmentation) to produce compact yet information-rich synthetic videos. The training objective combines a dataset distillation matching loss with a diversity loss, enabling end-to-end optimization that preserves temporal information while reducing both inter-sample and within-sample redundancies. Experiments across MiniUCF, HMDB51, SSv2, and K400 show state-of-the-art performance, with pronounced gains at higher IPC budgets, highlighting the method's practical impact for efficient video model training and evaluation.

Abstract

The rapid development of AI models has led to a growing emphasis on enhancing their capabilities for complex input data such as videos. While large-scale video datasets have been introduced to support this growth, the unique challenges of reducing redundancies in video \textbf{sets} have not been explored. Compared to image datasets or individual videos, video \textbf{sets} have a two-layer nested structure, where the outer layer is the collection of individual videos, and the inner layer contains the correlations among frame-level data points to provide temporal information. Video \textbf{sets} have two dimensions of redundancies: within-sample and inter-sample redundancies. Existing methods like key frame selection, dataset pruning or dataset distillation are not addressing the unique challenge of video sets since they aimed at reducing redundancies in only one of the dimensions. In this work, we are the first to study Video Set Distillation, which synthesizes optimized video data by jointly addressing within-sample and inter-sample redundancies. Our Information Diversification and Temporal Densification (IDTD) method jointly reduces redundancies across both dimensions. This is achieved through a Feature Pool and Feature Selectors mechanism to preserve inter-sample diversity, alongside a Temporal Fusor that maintains temporal information density within synthesized videos. Our method achieves state-of-the-art results in Video Dataset Distillation, paving the way for more effective redundancy reduction and efficient AI model training on video datasets.

Figures (5)

• Figure 1: The grand challenge of Video Set Distillation comes from its nature as a two-layer nested set. Each individual video is a set of image-level data points, and video set is a set of individual videos. It is critical to jointly optimize over both inter-sample redundancy and the within-sample redundancy. Furthermore, given the large variety of temporal length in a video set, the within-sample redundancies are largely different from each other, increasing the complexity of redundancy reduction.
• Figure 2: As shown on the left, Our IDTD approach jointly conduct the Information Diversification and the Temporal Densification in an end-to-end manner. On the top right, the Information Diversification part is shown in detail. The Feature Pool is a learnable variable and the Feature Selectors are learnable modules. On the bottom right, the Temporal Diversificaiton part is shown in detail. The Temporal Fusor is a learnable module. The Diversity loss enforces $K$ diverse segments to represent information from the original dataset, and the Temporal Fusor is optimized to integrate the diverse feature into a synthetic video instance driven by a dataset distillation matching loss. $K$ is a hyperparameter determining number of Feature Selectors for each synthetic instance. Our approach effectively balance the redundancy reduction between inter-sample dimension and within-sample dimension, while it keeps the temporal diverse information.
• Figure 3: We compared the trend of performance as number of frames of synthetic videos changes. Ours shows a clear growth as the number of frames increases, while VDSD dd12 have no significant increase.
• Figure 4: Redundancy Analysis. As shown in the diagram, our approach obtained higher performance gain where both inter-sample redundancy and within-sample redundancy are higher, indicating the effectiveness of joint optimization rather than naively discarding all temporal information followed by interpolation.
• Figure 5: Qualitative Results compared between image-level approach (Distribution Matching) and our IDTD approach jointly optimizaing within-sample and inter-sample class redundancy. Our approach preserves much more temporal information diversity. Class (a) is CleanAndJerk and Class(b) is playing violin.