PnLCalib: Sports Field Registration via Points and Lines Optimization

TL;DR

This work proposes an optimization-based calibration pipeline that leverages a 3D soccer field model and a predefined set of keypoints to overcome limitations of traditional search-based methods, and offers significant improvements in camera calibration precision and reliability.

Abstract

Camera calibration in broadcast sports videos presents numerous challenges for accurate sports field registration due to multiple camera angles, varying camera parameters, and frequent occlusions of the field. Traditional search-based methods depend on initial camera pose estimates, which can struggle in non-standard positions and dynamic environments. In response, we propose an optimization-based calibration pipeline that leverages a 3D soccer field model and a predefined set of keypoints to overcome these limitations. Our method also introduces a novel refinement module that improves initial calibration by using detected field lines in a non-linear optimization process. This approach outperforms existing techniques in both multi-view and single-view 3D camera calibration tasks, while maintaining competitive performance in homography estimation. Extensive experimentation on real-world soccer datasets, including SoccerNet-Calibration, WorldCup 2014, and TS-WorldCup, highlights the robustness and accuracy of our method across diverse broadcast scenarios. Our approach offers significant improvements in camera calibration precision and reliability.

Figures (6)

• Figure 1: Overview of our proposed framework.Top: Training data generation pipeline. Beginning with SoccerNet cioppa2022scaling annotations, we utilize field line extraction and ellipse fitting to establish a hierarchical structure for computing each set of keypoints. Bottom: Inference stage pipeline. The encoder-decoder networks produce heatmaps for keypoints and extremities of soccer field lines to extract their positions in the image space. The obtained keypoint set is augmented with intersections of lines generated by the second model to ensure a sufficient number of points. After initial calibration, our PnL refinement module is applied to further refine the calibration estimate by jointly using detected points and lines information.
• Figure 2: Definition of keypoint positions on a soccer field.Top: Distribution of points on a zenithal view, including all the relevant locations as a result of intersecting lines or curves in the field. $\mathcal{K}p$, $\mathcal{K}e$, $\mathcal{K}p_{1}$, $\mathcal{K}p_{2}$ and $\mathcal{K}p_{3}$ point sets are displayed in red, yellow, blue, pink and green points, respectively. Bottom: Given an external point, both $\mathcal{K}p_{1}$ and $\mathcal{K}p_{2}$ candidates are analytically derived, marked as blue and pink crosses, respectively.
• Figure 3: PnL refinement module. Top-Left: Let $\{\mathbf{p_d},\mathbf{q_d}\}$ be the visible extremities of a detected 3D soccer field line and $\{\mathbf{\bar{p}_d},\mathbf{\bar{q}_d}\}$ their projected 2D points in the image plane, which define the projected line $\mathbf{l_d}$ as well as the detected line projection given an initial calibration estimate. Top-Right:$d_1$ and $d_2$ represent the line-based reprojection error between $\mathbf{l_d}$ (in red) and $\mathbf{l}$ (in blue). Bottom: Soccer field's top-view. Estimated camera pose $\{\mathbf{R}, \mathbf{t}\}$ defines the camera plane $\mathbf{\Pi_c}$. Let $\{\mathbf{p},\mathbf{q}\}$ be the real extremities of a known 3D line given a field model, and $\mathbf{q_c}$ the corrected 3D point through the line-camera plane intersection. Note that $\mathbf{p}\equiv\mathbf{p_d}$ since the real line extremity is visible in the camera frame. The green area corresponds to the camera viewing cone.
• Figure 4: Qualitative results of the PnL refinement module.Top: Projection of soccer field lines with initial calibration estimate $\{\mathbf{K},\mathbf{R},\mathbf{t}\}$. Bottom: Projection of soccer field lines with refined calibration $\{\mathbf{K},\mathbf{R'},\mathbf{t'}\}$ through PnL refinement module.
• Figure 5: Qualitative results of our MV model on SN23-test.Left: Projection of soccer field lines and goal posts from world to image coordinates using predicted camera parameters. Blue lines correspond to segment projections, and colored points represent predicted keypoints (along with auxiliary points retrieved from line extremities detection). Right: SN23-test dataset annotations, where each soccer field line is delineated by a point set.