Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Aligning Model Properties via Conformal Risk Control

William Overman, Jacqueline Jil Vallon, Mohsen Bayati

TL;DR

A general procedure for converting queries for testing a given property of an aligned model as one belonging to a subset of functions suitable for use in a conformal risk control algorithm is developed.

Abstract

AI model alignment is crucial due to inadvertent biases in training data and the underspecified machine learning pipeline, where models with excellent test metrics may not meet end-user requirements. While post-training alignment via human feedback shows promise, these methods are often limited to generative AI settings where humans can interpret and provide feedback on model outputs. In traditional non-generative settings with numerical or categorical outputs, detecting misalignment through single-sample outputs remains challenging, and enforcing alignment during training requires repeating costly training processes. In this paper we consider an alternative strategy. We propose interpreting model alignment through property testing, defining an aligned model $f$ as one belonging to a subset $\mathcal{P}$ of functions that exhibit specific desired behaviors. We focus on post-processing a pre-trained model $f$ to better align with $\mathcal{P}$ using conformal risk control. Specifically, we develop a general procedure for converting queries for testing a given property $\mathcal{P}$ to a collection of loss functions suitable for use in a conformal risk control algorithm. We prove a probabilistic guarantee that the resulting conformal interval around $f$ contains a function approximately satisfying $\mathcal{P}$. We exhibit applications of our methodology on a collection of supervised learning datasets for (shape-constrained) properties such as monotonicity and concavity. The general procedure is flexible and can be applied to a wide range of desired properties. Finally, we prove that pre-trained models will always require alignment techniques even as model sizes or training data increase, as long as the training data contains even small biases.

Aligning Model Properties via Conformal Risk Control

TL;DR

A general procedure for converting queries for testing a given property of an aligned model as one belonging to a subset of functions suitable for use in a conformal risk control algorithm is developed.

Abstract

AI model alignment is crucial due to inadvertent biases in training data and the underspecified machine learning pipeline, where models with excellent test metrics may not meet end-user requirements. While post-training alignment via human feedback shows promise, these methods are often limited to generative AI settings where humans can interpret and provide feedback on model outputs. In traditional non-generative settings with numerical or categorical outputs, detecting misalignment through single-sample outputs remains challenging, and enforcing alignment during training requires repeating costly training processes. In this paper we consider an alternative strategy. We propose interpreting model alignment through property testing, defining an aligned model as one belonging to a subset of functions that exhibit specific desired behaviors. We focus on post-processing a pre-trained model to better align with using conformal risk control. Specifically, we develop a general procedure for converting queries for testing a given property to a collection of loss functions suitable for use in a conformal risk control algorithm. We prove a probabilistic guarantee that the resulting conformal interval around contains a function approximately satisfying . We exhibit applications of our methodology on a collection of supervised learning datasets for (shape-constrained) properties such as monotonicity and concavity. The general procedure is flexible and can be applied to a wide range of desired properties. Finally, we prove that pre-trained models will always require alignment techniques even as model sizes or training data increase, as long as the training data contains even small biases.
Paper Structure (39 sections, 4 theorems, 69 equations, 3 figures, 4 tables, 2 algorithms)

This paper contains 39 sections, 4 theorems, 69 equations, 3 figures, 4 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. Properties and Property Testing for Set-Valued Functions
  4. Example.
  5. Conformal prediction and conformal risk control
  6. Conformal property alignment
  7. Multi-lambda conformal risk control
  8. Conformal property alignment from proximity oblivious testers
  9. Example.
  10. Amplifying Detection Probability via Independent Calls.
  11. Examples
  12. Monotonicity
  13. Setup.
  14. Results.
  15. Concavity
  16. ...and 24 more sections

Key Result

Proposition 1

Assume that $L_i(\boldsymbol{\lambda})$ is non-increasing with respect to the partial ordering of $\boldsymbol{\Lambda}$ inherited from $\mathbb{R}^k$. Additionally, assume that $L_i(\boldsymbol{\lambda})$ is non-increasing with respect to $g(\boldsymbol{\lambda})$ for some strictly increasing funct

Figures (3)

  • Figure 1: Univariate partial dependence plot of unconstrained model. Risk control band for $\alpha=0.05$. Dashed line exemplifying Theorem 1 demonstrating existence of monotonically decreasing function falling within the conformal band on $0.975 > 1-\alpha$ fraction of the domain.
  • Figure 2: Random Feature model (N=5).
  • Figure 3: Random Feature model (N=5000).

Theorems & Definitions (20)

  • Definition 1: Satisfying and Accommodating a Property
  • Definition 2: $\varepsilon$-Faraway
  • Definition 3: One-Sided Error Tester for Set-Valued Functions
  • Definition 4: Proximity-Oblivious Tester for Set-Valued Functions
  • Definition 5: Construction of $F_{\boldsymbol{\lambda}}(X)$
  • Proposition 1
  • Definition 6: Loss Function Generated from a POT
  • Theorem 1
  • proof
  • Definition 7: $\varepsilon$-far
  • ...and 10 more