A Universal Protocol to Benchmark Camera Calibration for Sports

TL;DR

This work targets a core bottleneck in sports analytics: evaluating camera calibration without bias toward a single model like a field-plane homography. It introduces ProCC, a model-agnostic benchmarking protocol that uses semantic 3D field annotations and a JaC$_{\tau}$ metric to assess how well estimated camera parameters reproject known 3D objects into images. The approach reveals that homography-based ground truth inadequately captures calibration quality and that richer camera models, notably with radial distortion, yield more accurate results on broadcast cameras. While demonstrated in soccer datasets (WC14, CARWC, SoccerNet), ProCC provides a general framework for fair calibration benchmarking across sports and camera setups, enabling better optimization of calibration pipelines and multi-view calibration strategies. This protocol thus establishes a path toward higher accuracy standards in camera calibration for real-world sports applications.

Abstract

Camera calibration is a crucial component in the realm of sports analytics, as it serves as the foundation to extract 3D information out of the broadcast images. Despite the significance of camera calibration research in sports analytics, progress is impeded by outdated benchmarking criteria. Indeed, the annotation data and evaluation metrics provided by most currently available benchmarks strongly favor and incite the development of sports field registration methods, i.e. methods estimating homographies that map the sports field plane to the image plane. However, such homography-based methods are doomed to overlook the broader capabilities of camera calibration in bridging the 3D world to the image. In particular, real-world non-planar sports field elements (such as goals, corner flags, baskets, ...) and image distortion caused by broadcast camera lenses are out of the scope of sports field registration methods. To overcome these limitations, we designed a new benchmarking protocol, named ProCC, based on two principles: (1) the protocol should be agnostic to the camera model chosen for a camera calibration method, and (2) the protocol should fairly evaluate camera calibration methods using the reprojection of arbitrary yet accurately known 3D objects. Indirectly, we also provide insights into the metric used in SoccerNet-calibration, which solely relies on image annotation data of viewed 3D objects as ground truth, thus implementing our protocol. With experiments on the World Cup 2014, CARWC, and SoccerNet datasets, we show that our benchmarking protocol provides fairer evaluations of camera calibration methods. By defining our requirements for proper benchmarking, we hope to pave the way for a new stage in camera calibration for sports applications with high accuracy standards.

Figures (5)

• Figure 1: Illustration of a successful camera calibration. Lines superimposed in blue are obtained by projecting the markings of a soccer field. A perfect alignment is mandatory to enable high-precision applications such as offside position assessment (in red, a parallel to the goal line to decide on an offside situation). This paper proposes a new protocol, named ProCC, that has two advantages: (1) it is applicable to any sport, and (2) its evaluation is based on a new metric which is agnostic to the chosen camera type and model.
• Figure 2: Visualization of the $\text{IoU}_{\text{whole}}$ and $\text{IoU}_{\text{part}}$ metrics. Illustration taken from Sha2020EndtoEnd (© IEEE, 2020).
• Figure 3: Illustration of annotations on a SoccerNet-v3 Cioppa2022Scaling image as used for the camera calibration challenge. As shown in the zoomed snapshot, annotations consist of points and the labels of the objects they belong to.
• Figure 4: Comparison of reprojected field elements based on different types of annotations: in the first column, the WC14 homographies are used to project the soccer field model. The second column corresponds to the CARWC annotations. Both these annotations fail to provide correct reprojections. Finally, the third column displays results obtained with the Xeebra product.
• Figure 5: Illustration of the disagreement between estimations from two camera models: the ground-truth homography (see the red wireframes) and the richer model combining the pinhole and radial distortion used in Xeebra (see the green wireframes). The small difference between the two reprojected wireframes is misleading, as the superimposition of contour plots on the field shows that some parts of the sports field are seen over 2.5 meters apart by these two camera models. We have also plotted the contour line for a difference of 50 cm (see the dashed lines), which is the minimal accuracy standard demanded by the FIFA for Offside Technologies, to highlight why methods developed with older calibration protocols such as the WC14 or CARWC would fail to meet professional requirements.