Spectrum AUC Difference (SAUCD): Human-aligned 3D Shape Evaluation

TL;DR

An analytic metric named Spectrum Area Under the Curve Difference (SAUCD) is proposed that demonstrates better consistency with human evaluation, and outperforms previous 3D mesh metrics.

Abstract

Existing 3D mesh shape evaluation metrics mainly focus on the overall shape but are usually less sensitive to local details. This makes them inconsistent with human evaluation, as human perception cares about both overall and detailed shape. In this paper, we propose an analytic metric named Spectrum Area Under the Curve Difference (SAUCD) that demonstrates better consistency with human evaluation. To compare the difference between two shapes, we first transform the 3D mesh to the spectrum domain using the discrete Laplace-Beltrami operator and Fourier transform. Then, we calculate the Area Under the Curve (AUC) difference between the two spectrums, so that each frequency band that captures either the overall or detailed shape is equitably considered. Taking human sensitivity across frequency bands into account, we further extend our metric by learning suitable weights for each frequency band which better aligns with human perception. To measure the performance of SAUCD, we build a 3D mesh evaluation dataset called Shape Grading, along with manual annotations from more than 800 subjects. By measuring the correlation between our metric and human evaluation, we demonstrate that SAUCD is well aligned with human evaluation, and outperforms previous 3D mesh metrics.

Figures (15)

• Figure 1: An example of how previous spatial domain 3D shape metrics (Chamfer Distance borgefors1984CD and UHD wu2020uhd) deviate from human evaluation. we create Mesh $A$ by adding a small pose error to the ground truth mesh, and by applying a large smoothing kernel to ground truth, we create Mesh $B$. Contrary to human perception, previous spatial domain metrics evaluate Mesh $B$ better than Mesh $A$. This indicates that while they are sensitive to general shape differences, they tend to overlook high-frequency details. Note that different metrics use different units of measurement.
• Figure 2: Our SAUCD metric is designed as follows: A. We use mesh Fourier Transform to analyze the spectrums of test and ground truth mesh. B. We compare the difference between two spectrum curves by calculating the Area Under the Curve (AUC) difference. C. We further extend our metric by multiplying the AUC difference with a learnable weight to capture human sensitivity in each frequency band.
• Figure 3: Variables defined in our discrete Laplace-Beltrami operator design.
• Figure 4: Spectrum Area Under the Curve Difference. We design our metric using the AUC difference of the spectrums. The blue curve and red curve are the test and ground truth mesh spectrum, respectively. The purple area in the last graph is the Spectrum AUC Difference. Please find details in \ref{['sec:sad']}.
• Figure 5: An example of mesh spectrum curve: We do mesh Fourier transform on the "Origin" mesh and show the spectrum in the left graph. The $\lambda$-axis is the eigenvalues of the DLBO matrix, the larger the higher frequency. We also show how mesh changes when gradually removing high-frequency information (mesh A to G). The frequency bands of the meshes are shown as the colored arrows in the left graph.