A Synthetic Benchmarking Pipeline to Compare Camera Calibration Algorithms

TL;DR

SynthCal addresses the lack of ground-truth data for camera calibration by delivering a fully synthetic benchmarking pipeline that generates calibration images with known intrinsic and extrinsic parameters. The framework renders four calibration-pattern types across two camera configurations and lighting conditions, enabling precise evaluation using $RMSE$ and $RPE_{RMS}$ for both monocular and multi-view setups. By comparing Zhang, Tsai, and Bouguet methods across patterns and configurations, the work reveals patterns (e.g., Charuco) and algorithms (e.g., Bouguet) that provide robust performance under standard conditions, while highlighting the strengths of Zhang in noisier environments. These findings offer practical guidance for pattern choice and calibration algorithm selection and establish a reproducible benchmark for future camera calibration research.

Abstract

Accurate camera calibration is crucial for various computer vision applications. However, measuring calibration accuracy in the real world is challenging due to the lack of datasets with ground truth to evaluate them. In this paper, we present SynthCal, a synthetic camera calibration benchmarking pipeline that generates images of calibration patterns to measure and enable accurate quantification of calibration algorithm performance in camera parameter estimation. We present a SynthCal generated calibration dataset with four common patterns, two camera types, and two environments with varying view, distortion, lighting, and noise levels for both monocular and multi-camera systems. The dataset evaluates both single and multi-view calibration algorithms by measuring re-projection and root-mean-square errors for identical patterns and camera settings. Additionally, we analyze the significance of different patterns using different calibration configurations. The experimental results demonstrate the effectiveness of SynthCal in evaluating various calibration algorithms and patterns.

Figures (4)

• Figure 1: SynthCal pipeline to generate calibration dataset from a set of input attributes: Pattern attributes, camera intrinsic, distortion, extrinsic matrix. The accuracy is then evaluated using $RMSE$ and $RPE_{RMS}$ for monocular cameras.
• Figure 2: (a) 9 $\times$ 12 Charuco pattern, (b) 10 $\times$ 10 Symmetric circle grid, (c) 9 $\times$ 10 Asymmetric circle grid, (d) 9 $\times$ 12 Checkerboard pattern.
• Figure 3: (a, b) Original clean and noisy capture, undistorted render using the camera parameters predicted by (c, d) Zhang, (e, f) Tsai, (g, h) Bouguet method.
• Figure 4: Using SynthCal pipeline with DMCB ray2023selecting and EasyMocap dong2021fast to estimate 3D pose from multiple view points