Learning Monotonic Probabilities with a Generative Cost Model

TL;DR

The paper addresses learning monotonic probabilities by recasting the problem as a latent-cost modeling task. It introduces two generative frameworks, the Generative Cost Model (GCM) for strict monotonicity and the Implicit Generative Cost Model (IGCM) for implicit monotonicity, both trained via variational inference and leveraging a latent cost variable $\boldsymbol{\mathbf{c}}$. The authors demonstrate that predicting $\Pr(y|\boldsymbol{x},\boldsymbol{r})$ can be achieved through the latent relation $\boldsymbol{c} \prec \boldsymbol{r}$, enabling robust monotonic behavior without constructing monotonic functions explicitly; IGCM further handles practical, non-strict monotonicity via a kernel variable $\boldsymbol{k}$. Empirical results on simulated quantile regression and six public datasets show that GCM and IGCM substantially outperform traditional monotonic models, with IGCM offering advantages when monotonicity is present only implicitly. The approach provides a scalable, probabilistic alternative to monotonic neural networks and opens avenues for more flexible monotonic modeling in complex data settings.

Abstract

In many machine learning tasks, it is often necessary for the relationship between input and output variables to be monotonic, including both strictly monotonic and implicitly monotonic relationships. Traditional methods for maintaining monotonicity mainly rely on construction or regularization techniques, whereas this paper shows that the issue of strict monotonic probability can be viewed as a partial order between an observable revenue variable and a latent cost variable. This perspective enables us to reformulate the monotonicity challenge into modeling the latent cost variable. To tackle this, we introduce a generative network for the latent cost variable, termed the Generative Cost Model (GCM), which inherently addresses the strict monotonic problem, and propose the Implicit Generative Cost Model (IGCM) to address the implicit monotonic problem. We further validate our approach with a numerical simulation of quantile regression and conduct multiple experiments on public datasets, showing that our method significantly outperforms existing monotonic modeling techniques. The code for our experiments can be found at https://github.com/tyxaaron/GCM.

Figures (6)

• Figure 1: An example of partial order between vectors, where $\bm r_1 \prec \bm r_2$ and $\bm r_1 \prec \bm r_3$. The partial order between $\bm r_2$ and $\bm r_3$ is not determined.
• Figure 2: An illustration of the probability distribution functions for three random variables $\textnormal{y}_1$, $\textnormal{y}_2$, and $\textnormal{y}_3$, given that $\textnormal{y}_1 \prec_{\rm r.v.} \textnormal{y}_2$ and $\textnormal{y}_1 \prec_{\rm r.v.} \textnormal{y}_3$, is presented. It is apparent that ${\rm epi}F_1$ is a subset of both ${\rm epi}F_2$ and ${\rm epi}F_3$. However, there is no containment relationship between the sets ${\rm epi}F_2$ and ${\rm epi}F_3$, indicating that the variables $\textnormal{y}_2$ and $\textnormal{y}_3$ cannot be directly compared.
• Figure 3: Within the density contour plot for the cost variable $\boldsymbol{\mathbf{c}}$, the shaded area represents the event $\boldsymbol{\mathbf{c}} \prec \boldsymbol{\mathbf{r}}$. This event indicates that the probability of the cost variable $\boldsymbol{\mathbf{c}}$ falling within this shaded region is expressed as $\Pr(\boldsymbol{\mathbf{c}} \prec \boldsymbol{\mathbf{r}})$. Thus, for any vectors $\bm r_1 \prec \bm r_2$, it follows that $\Pr(\boldsymbol{\mathbf{c}} \prec \bm r_1) < \Pr(\boldsymbol{\mathbf{c}} \prec \bm r_2)$.
• Figure 4: The probabilistic graphical model of the generative cost model ensures a monotonic conditional probability $p(\textnormal{y}|\boldsymbol{\mathbf{x}}, \boldsymbol{\mathbf{r}})$ with respect to $\boldsymbol{\mathbf{r}}$. Within this diagram, observable variables are depicted as gray nodes, whereas latent variables are shown as white nodes. Solid arrows illustrate the generative model $p_\theta$, and the dashed arrow indicates the variational model $q_\phi$. The thick arrows depict the estimator for $p(\textnormal{y} | \boldsymbol{\mathbf{x}}, \boldsymbol{\mathbf{r}})$.
• Figure 5: The probability graphical model for the implicit generative cost model (IGCM), which doses not ensure strict monotonic between $\boldsymbol{\mathbf{r}}$ and $\textnormal{y}$, but introduces monotonicity from $\boldsymbol{\mathbf{k}}$ to $\boldsymbol{\mathbf{r}}$ and from $\boldsymbol{\mathbf{k}}$ to $\textnormal{y}$. The thick arrows represent the path for the prediction of $p(\textnormal{y}|\boldsymbol{\mathbf{x}}, \boldsymbol{\mathbf{r}})$.

Theorems & Definitions (8)

• Definition 2.1: Partial Order in Vector Space
• Definition 2.2: Strict Order in Vector Space
• Definition 2.3: First-Order Stochastic Dominance hadar1969rules
• Definition 2.4: Monotonic Conditional Probability
• Lemma 4.1
• Lemma 4.2
• proof
• proof