Breakdance Video classification in the age of Generative AI

TL;DR

This work addresses breakdance move classification using modern video foundation models by comparing encoder-based approaches (ViViT, VideoMAE, ImageBind) with a decoder-based LLM (Qwen-2.5-VL) on the BRACE dataset. It demonstrates that encoder-based models achieve superior predictive performance for this high-speed sport, with ImageBind offering the best generalization in embedding space. An extensive decoder finetuning ablation reveals that higher LoRA ranks, non-greedy decoding, and carefully chosen label descriptions can improve performance, though these effects depend on model complexity. The study provides practical guidance for applying video foundation models to niche sports and highlights the value of cross-modal, encoder-derived representations, while releasing code and embeddings to foster reproducibility.

Abstract

Large Vision Language models have seen huge application in several sports use-cases recently. Most of these works have been targeted towards a limited subset of popular sports like soccer, cricket, basketball etc; focusing on generative tasks like visual question answering, highlight generation. This work analyzes the applicability of the modern video foundation models (both encoder and decoder) for a very niche but hugely popular dance sports - breakdance. Our results show that Video Encoder models continue to outperform state-of-the-art Video Language Models for prediction tasks. We provide insights on how to choose the encoder model and provide a thorough analysis into the workings of a finetuned decoder model for breakdance video classification.

Figures (14)

• Figure 1: Breakdance Video and time stamped tags in Eluvio Video Intelligence Editor (EVIE)
• Figure 2: The label distribution. x - axis move duration in seconds, y - axis frequency of video segments. The move durations (in secs) for a) footwork = 11.5 $\pm$ 6.95 sec, b) powermove = 9.85 $\pm$ 5.9 sec, c) toprock = 7.01 $\pm$ 4.01 sec.
• Figure 3: Schematic representation of the breakdance video's moving window prediction.
• Figure 4: Encoder Model Architecture.
• Figure 5: Multiclass Hinge loss for linear mapping $f(\mathbf{x}) = \mathbf{w}^\top \mathbf{x}$. Sample $(\mathbf{x}_i,y_i)$ lying inside the margin is linearly penalized using slack variable $\xi_i$.