Transport Novelty Distance: A Distributional Metric for Evaluating Material Generative Models

TL;DR

The paper introduces Transport Novelty Distance (TNovD), a distribution-level metric based on Optimal Transport to jointly quantify quality and novelty in material generative models. It uses a Graph Neural Network to embed crystal structures into a discriminative feature space and computes a customized OT-based coupling that balances memorization and quality via a threshold τ and hyperparameter M. The authors validate TNovD on toy experiments, MP20 validation, and the WBM dataset, showing that it can detect memorized and low-quality data while comparing various generative models. They also discuss limitations, such as stability-awareness, and emphasize the metric's domain-agnostic potential for extension to images and molecules.

Abstract

Recent advances in generative machine learning have opened new possibilities for the discovery and design of novel materials. However, as these models become more sophisticated, the need for rigorous and meaningful evaluation metrics has grown. Existing evaluation approaches often fail to capture both the quality and novelty of generated structures, limiting our ability to assess true generative performance. In this paper, we introduce the Transport Novelty Distance (TNovD) to judge generative models used for materials discovery jointly by the quality and novelty of the generated materials. Based on ideas from Optimal Transport theory, TNovD uses a coupling between the features of the training and generated sets, which is refined into a quality and memorization regime by a threshold. The features are generated from crystal structures using a graph neural network that is trained to distinguish between materials, their augmented counterparts, and differently sized supercells using contrastive learning. We evaluate our proposed metric on typical toy experiments relevant for crystal structure prediction, including memorization, noise injection and lattice deformations. Additionally, we validate the TNovD on the MP20 validation set and the WBM substitution dataset, demonstrating that it is capable of detecting both memorized and low-quality material data. We also benchmark the performance of several popular material generative models. While introduced for materials, our TNovD framework is domain-agnostic and can be adapted for other areas, such as images and molecules.

Figures (7)

• Figure 1: Graphical explanation of the complete workflow used to derive the Transport Novelty Distance between generated sets of materials.
• Figure 2: Visualization of an OT coupling between two discrete distributions. The connecting lines indicate the coupling, i.e., the sample assignments.
• Figure 3: (a): True uniform distribution on the circle. (b) to (d): Generated samples for increasing smoothing values $\sigma$. (e) and (f): Wasserstein distance between validation and generated samples and Transport Novelty Distance, both for varying smoothing values $\sigma$.
• Figure 4: Illustration of optimal-transport matchings between training data $P_D$ (green) and generated samples $P_G$ (blue). Irregular mappings distinguish good vs. low-quality generations; distorted vs. aligned shapes indicate poor vs. faithful model geometry.
• Figure 5: Different toy experiments performed to test the TNovD behavior for specific test cases.