Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Preferential Normalizing Flows

Petrus Mikkola, Luigi Acerbi, Arto Klami

TL;DR

This work introduces a method for eliciting the expert's belief density as a normalizing flow based solely on preferential questions such as comparing or ranking alternatives, and shows empirically that the belief density can be inferred as the function-space maximum a posteriori estimate.

Abstract

Eliciting a high-dimensional probability distribution from an expert via noisy judgments is notoriously challenging, yet useful for many applications, such as prior elicitation and reward modeling. We introduce a method for eliciting the expert's belief density as a normalizing flow based solely on preferential questions such as comparing or ranking alternatives. This allows eliciting in principle arbitrarily flexible densities, but flow estimation is susceptible to the challenge of collapsing or diverging probability mass that makes it difficult in practice. We tackle this problem by introducing a novel functional prior for the flow, motivated by a decision-theoretic argument, and show empirically that the belief density can be inferred as the function-space maximum a posteriori estimate. We demonstrate our method by eliciting multivariate belief densities of simulated experts, including the prior belief of a general-purpose large language model over a real-world dataset.

Preferential Normalizing Flows

TL;DR

This work introduces a method for eliciting the expert's belief density as a normalizing flow based solely on preferential questions such as comparing or ranking alternatives, and shows empirically that the belief density can be inferred as the function-space maximum a posteriori estimate.

Abstract

Eliciting a high-dimensional probability distribution from an expert via noisy judgments is notoriously challenging, yet useful for many applications, such as prior elicitation and reward modeling. We introduce a method for eliciting the expert's belief density as a normalizing flow based solely on preferential questions such as comparing or ranking alternatives. This allows eliciting in principle arbitrarily flexible densities, but flow estimation is susceptible to the challenge of collapsing or diverging probability mass that makes it difficult in practice. We tackle this problem by introducing a novel functional prior for the flow, motivated by a decision-theoretic argument, and show empirically that the belief density can be inferred as the function-space maximum a posteriori estimate. We demonstrate our method by eliciting multivariate belief densities of simulated experts, including the prior belief of a general-purpose large language model over a real-world dataset.

Paper Structure

This paper contains 32 sections, 7 theorems, 23 equations, 18 figures, 5 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Why learning the density from preferential data is challenging?
  3. Random utility model with exponentially distributed noises
  4. The $k$-wise winner
  5. The $k$-wise winner distribution as a tilted and tempered belief density
  6. Belief density as normalizing flow
  7. Function-space Bayesian inference
  8. Empirical functional prior
  9. Function-space maximum a posteriori
  10. Experiments
  11. Setup
  12. Evaluation
  13. Synthetic tasks
  14. Expert elicitation with LLM as the expert
  15. Ablation study
  16. ...and 17 more sections

Key Result

Proposition 2.1

Let $p_{\star}$ be the expert's belief density. For $k \geq 2$, let $\mathcal{D}_{\textrm{rank}} := \{\mathbf{x}_1 \succ \mathbf{x}_2 \succ ... \succ \mathbf{x}_{k}\}$ be a $k$-wise ranking (see Definition k-wise_ranking_def). If $W \sim \textrm{Gumbel}(0,\beta)$, then for any positive monotonic tra

Figures (18)

  • Figure 1: Illustration of belief densities elicited from preferential ranking data by a normalizing flow (contour: true density; heatmap: estimated flow; red: preferred points; blue: non-preferred points). (a)-(b): Typical failure modes of collapsing and diverging mass, when training a flow with just $n = 10$ rankings. (c)-(d): The proposed functional prior resolves the issues, and already with 10 rankings we can learn the correct belief density, matching the result of the flow trained on larger data.
  • Figure 2: (a) The $k$-wise winner distribution converges to the belief density as $k \rightarrow \infty$. (b) The $k$-wise winner distribution can be approximated by a tempered belief density. For example, the tempered belief density with an exponent $1/5$ approximates well the pairwise winner distribution.
  • Figure 3: Cross-plot of selected variables of the estimated flow in the Abalone (left) and LLM knowledge elicitation experiment (middle), and the marginal density of the same variables for the ground truth density in the LLM experiment (right). See Figures \ref{['fig-llmexp-full']} and \ref{['fig:abalonefull']} for other variables.
  • Figure 4: Illustration of belief densities elicited from pairwise comparisons by a normalizing flow.
  • Figure C.1: Full result plot for the modified Abalone7D experiment, where the target (unnormalized) belief density corresponds to the abalone age.
  • ...and 13 more figures

Theorems & Definitions (18)

  • Proposition 2.1: Unidentifiability of a noiseless RUM
  • Definition 3.1: choice set
  • Definition 3.2: $k$-wise comparison
  • Definition 3.3: $k$-wise ranking
  • Definition 3.4: $k$-wise winner
  • Proposition 3.5
  • proof
  • Theorem 3.6
  • proof
  • Example A.1
  • ...and 8 more