Risk-based Calibration for Generative Classifiers

TL;DR

This work tackles the gap between generative classifiers and supervised classification performance by proposing risk-based calibration (RC), an iterative procedure that refines the joint distribution $p_h(\mathbf{x},y)$ using a soft 0-1 loss $l_{s01}(h,(\boldsymbol{x},y)) = 1-p_h(y|\boldsymbol{x})$. RC updates the sufficient statistics via $\boldsymbol{s} \leftarrow \boldsymbol{s} + s(\boldsymbol{x},y) - s(\boldsymbol{x},h)$, and then recomputes parameters with existing closed-form learning, ensuring validity without extra constraint handling. The authors demonstrate RC on two representative generative classifiers, Naïve Bayes and Quadratic Discriminant Analysis, across 20 diverse datasets, consistently achieving lower training and generalization errors than standard ML and gradient-based methods, with favorable convergence properties. RC thus offers a practical, scalable way to bridge generative modeling with discriminative objectives, and the approach is reinforced by connections to Discriminative Frequency Estimate and the TM algorithm, along with open-source code for reproducibility.

Abstract

Generative classifiers are constructed on the basis of a joint probability distribution and are typically learned using closed-form procedures that rely on data statistics and maximize scores related to data fitting. However, these scores are not directly linked to supervised classification metrics such as the error, i.e., the expected 0-1 loss. To address this limitation, we propose a learning procedure called risk-based calibration (RC) that iteratively refines the generative classifier by adjusting its joint probability distribution according to the 0-1 loss in training samples. This is achieved by reinforcing data statistics associated with the true classes while weakening those of incorrect classes. As a result, the classifier progressively assigns higher probability to the correct labels, improving its training error. Results on 20 heterogeneous datasets using both naïve Bayes and quadratic discriminant analysis show that RC significantly outperforms closed-form learning procedures in terms of both training error and generalization error. In this way, RC bridges the gap between traditional generative approaches and learning procedures guided by performance measures, ensuring a closer alignment with supervised classification objectives.

Figures (1)

• Figure 1: Example of the initialization and first iteration of RC using a simple model with two parameters $\mu_1$ and $\mu_2$ for unidimensional instances with three training samples