PALATE: Peculiar Application of the Law of Total Expectation to Enhance the Evaluation of Deep Generative Models

TL;DR

PALATE tackles the challenge of holistically evaluating deep generative models by incorporating memorization detection into fidelity-diversity-novelty assessment. It builds on the law of total expectation to blend a baseline metric with a novelty-aware component, producing a single metric that flags data-copying tendencies while preserving computational efficiency. Implemented on top of the DMMD baseline with DINOv2 embeddings, PALATE demonstrates competitive or superior performance to state-of-the-art metrics on CIFAR-10 and ImageNet, with reduced computational demands and scalability to large datasets. These results suggest PALATE offers a practical, theoretically grounded framework for robust DGM evaluation and memorization detection in real-world settings.

Abstract

Deep generative models (DGMs) have caused a paradigm shift in the field of machine learning, yielding noteworthy advancements in domains such as image synthesis, natural language processing, and other related areas. However, a comprehensive evaluation of these models that accounts for the trichotomy between fidelity, diversity, and novelty in generated samples remains a formidable challenge. A recently introduced solution that has emerged as a promising approach in this regard is the Feature Likelihood Divergence (FLD), a method that offers a theoretically motivated practical tool, yet also exhibits some computational challenges. In this paper, we propose PALATE, a novel enhancement to the evaluation of DGMs that addresses limitations of existing metrics. Our approach is based on a peculiar application of the law of total expectation to random variables representing accessible real data. When combined with the MMD baseline metric and DINOv2 feature extractor, PALATE offers a holistic evaluation framework that matches or surpasses state-of-the-art solutions while providing superior computational efficiency and scalability to large-scale datasets. Through a series of experiments, we demonstrate the effectiveness of the PALATE enhancement, contributing a computationally efficient, holistic evaluation approach that advances the field of DGMs assessment, especially in detecting sample memorization and evaluating generalization capabilities.

Figures (7)

• Figure 1: Comparison of the effects of different transformations, applied to samples generated by PFGM++, on $M_\text{PALATE{}}$ and FLD (corresponding values for other models are provided for reference). Left: Nearly imperceptible transformations. Right: Large transformations.
• Figure 2: Capability of $M_\text{PALATE{}}$ (our) and FLD to capture sample diversity in two experimental settings. Left: Varying the number of classes while maintaining a fixed total sample size of $10000$ by adjusting the duplication of $1000$ fixed samples per class. Right: Varying the number of unique samples per class, with equal replication across classes to maintain class balance and a total sample size of $10000$.
• Figure 3: Capability of ${\text{PALATE}}$ (our) to capture sample novelty, compared to different memorization metrics. The $y$-axis for our metric has been inverted for visual consistency.
• Figure 4: $M_{\text{PALATE}}$ and FLD evaluated on the mixture of generated and training images from CIFAR-10, ranging from $0\%$ train (purely generated) to $100\%$ train (purely training). Since FLD eventually "blows up," its y-axis is plotted on a logarithmic scale.
• Figure 5: Left: Evaluation of $M_{\text{PALATE}}$ and FLD for different sample sizes on the CIFAR-10 dataset. Solid lines represent the mean values calculated over 10 runs with different seeds, while the shaded regions show the mean values $\pm$ standard deviation. Right: Computing time comparison for both metrics on ImageNet. The FLD plot is truncated at a sample size of $20000$ due to its memory inefficiency on larger datasets.

Theorems & Definitions (3)

• Theorem 1
• Definition 1
• Definition 2