FSBench: A Figure Skating Benchmark for Advancing Artistic Sports Understanding

TL;DR

This paper introduces FSAnno, a fine-grained, multi-modal dataset for figure skating that captures both technical elements and artistic expression, and FSBench, a structured benchmark with FSBench-Text and FSBench-Motion to evaluate multi-task understanding. It presents SkateLLM, an instruction-tuned, coordinated multi-modal model built on FSAnno data to better interpret motion, GOE-based scoring, and performance commentary, using AutoDQ for nuanced open-ended evaluation. Experiments reveal current LLMs have substantial gaps in artistic-sports understanding, but targeted instruction tuning on FSAnno yields meaningful improvements, validating the dataset-benchmark pair as a tool to advance holistic figure skating comprehension. The work lays a foundation for robust, multi-task AI systems capable of analyzing and commenting on complex artistic sports, with broad implications for education, coaching, and benchmarking in non-ball, artistry-driven domains.

Abstract

Figure skating, known as the "Art on Ice," is among the most artistic sports, challenging to understand due to its blend of technical elements (like jumps and spins) and overall artistic expression. Existing figure skating datasets mainly focus on single tasks, such as action recognition or scoring, lacking comprehensive annotations for both technical and artistic evaluation. Current sports research is largely centered on ball games, with limited relevance to artistic sports like figure skating. To address this, we introduce FSAnno, a large-scale dataset advancing artistic sports understanding through figure skating. FSAnno includes an open-access training and test dataset, alongside a benchmark dataset, FSBench, for fair model evaluation. FSBench consists of FSBench-Text, with multiple-choice questions and explanations, and FSBench-Motion, containing multimodal data and Question and Answer (QA) pairs, supporting tasks from technical analysis to performance commentary. Initial tests on FSBench reveal significant limitations in existing models' understanding of artistic sports. We hope FSBench will become a key tool for evaluating and enhancing model comprehension of figure skating.

Figures (5)

• Figure 1: To systematically assess whether models understand figure skating which is an artistic sport, FSBench sources its data from multi-level competitions, incorporating multimodal data, multidimensional annotations, and diverse evaluation protocols. The tasks are structured with multiple roles and levels, ranging from prior knowledge testing, single action recognition, single action assessment, and commentary to the evaluation of entire performances. Additionally, to better evaluate the artistic dimension, our annotations include artistry-related scores, along with corresponding descriptions in the ground truth for both single action assessment and overall performance evaluation tasks. Detailed artistic expression information, such as emotional expression, is usually included in commentary annotations, as shown in the bottom-right corner.
• Figure 2: Data Statistics. (a) The figure skating elements are divided into three main categories: Jump, Spin, and Sequence, which are further divided into 20 subcategories. The bar chart shows the category distribution. (b) The pie chart shows the distribution of FSBench elements by gender; Men Short Program (MS), Men Free Skating (MF), Women Short Program (WS), Women Free Skating (WF); short program and free skate; and the positive and negative values of the $GOE$ scores. (c) The stacked bar chart shows the distribution of $GOE$ scores across different program types.
• Figure 3: Our annotations are derived from three sources: official judging reports, visual information, and audio commentary. These three official sources provide objective, multi-dimensional, fine-grained, and multi-task-oriented annotations for FSAnno and FSBench.
• Figure 4: Compared to other large language models, SkateLLM's descriptions of figure skating elements are more focused on the technical movements themselves and artistic evaluations. Additionally, based on the more professional captions it generates, other LLMs (such as GPT-4) have a higher probability of correctly inferring the category of the element.
• Figure 5: Evaluation pipeline using AutoDQ. For event extraction and cross-checking, we use GPT-3.5-turbo. These results can support more fine-grained evaluation.