WTMAD-4: A Fair Weighting Scheme for GMTKN55

Abstract

The GMTKN55 data set is a collection of standard benchmarks used in molecular quantum chemistry that spans small- and large-molecule thermochemistry, reaction barriers, and non-covalent interactions. Herein, we identify a flaw in the weighted mean absolute deviation (WTMAD) definitions commonly used to quantify performance of various electronic-structure methods for the GMTKN55 set, which under-weight some of its component benchmarks by orders of magnitude. A new WTMAD-4 metric is proposed, based on typical errors observed for a set of ten minimally empirical dispersion-corrected density-functional approximations (DFAs), ensuring fair treatment across all benchmarks. The performance of 115 DFAs is then reassessed using WTMAD-4 and we highlight a literature example where a DFA parametrised by minimising WTMAD-2 underperforms for benchmarks marginalised by that metric.

Figures (2)

• Figure 1: Smoothed histograms showing the percent contributions of each GMTKN55 subset to the various WTMAD-$N$ values for 115 DC-DFAs. The WTMAD-4 weights were obtained from Eq. \ref{['eq:weights']} using the mean MAD values from ten minimally empirical functionals.
• Figure 2: Box-and-whisker plots showing the percent contributions of each GMTKN55 subset to the various WTMAD-$N$ values for 115 DC-DFAs. For each WTMAD-$N$, the median is shown with a white line, the coloured boxes span the interquartile range (IQR) from the 25% to 75% quantiles. Whiskers extend to 1.5$\times$ the IQR, beyond which outliers are shown as individual points. The 98% quantile is indicated with a dashed line. The WTMAD-4 weights were obtained from Eq. \ref{['eq:weights']} using the mean MAD values from ten minimally empirical functionals.