FMAC: a Fair Fiducial Marker Accuracy Comparison Software

TL;DR

FMAC addresses fair comparison of fiducial-marker pose estimation by generating large sets of high-fidelity synthetic images that respect camera intrinsics and distortion. It uses low-discrepancy sampling of the six-DOF pose space to study correlations across 36 DOF–error pairings and applies the framework to four markers (ArUco, AprilTag, STag, TopoTag), quantifying translations and rotations and their detection-rate dependencies. The findings show TopoTag offers strong translational accuracy while AprilTag provides superior angular accuracy, with notable detection-rate variations across range; the authors provide an open-source FMAC pipeline for reproducible benchmarking. This work enables standardized, repeatable pose-accuracy assessments for camera-marker systems in real-world settings and supports systematic exploration of how imaging parameters affect pose estimation.

Abstract

This paper presents a method for carrying fair comparisons of the accuracy of pose estimation using fiducial markers. These comparisons rely on large sets of high-fidelity synthetic images enabling deep exploration of the 6 degrees of freedom. A low-discrepancy sampling of the space allows to check the correlations between each degree of freedom and the pose errors by plotting the 36 pairs of combinations. The images are rendered using a physically based ray tracing code that has been specifically developed to use the standard calibration coefficients of any camera directly. The software reproduces image distortions, defocus and diffraction blur. Furthermore, sub-pixel sampling is applied to sharp edges to enhance the fidelity of the rendered image. After introducing the rendering algorithm and its experimental validation, the paper proposes a method for evaluating the pose accuracy. This method is applied to well-known markers, revealing their strengths and weaknesses for pose estimation. The code is open source and available on GitHub.

Figures (9)

• Figure 1: Synthetic images with different fiducial markers at the same poses (overlay of 21 images).
• Figure 2: Illustration of pixels and lens sampling.
• Figure 3: Overlay of actual and synthetic images from the OpenCV calibration dataset. The pixels colored in red indicate that the actual intensity is over the synthetic intensity. The pixels colored in cyan show the opposite errors. There is no difference between images if the pixels are white, black, or gray.
• Figure 4: Comparison of defocus aberrations on the Matlab's calibration dataset. The bottom zooms show details that are in focus. The top zooms correspond to points that are beyond the depth of field limit.
• Figure 5: Errors between the estimated poses of an ArUco marker and the actual pose for the six degrees of freedom. The images are rendered using the Logitech HD Webcam C270 parameters. The horizontal dashed lines represent the mean of the errors and the dotted lines the mean plus or minus the standard deviation.