Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Learning Parametric Distributions from Samples and Preferences

Marc Jourdan, Gizem Yüce, Nicolas Flammarion

TL;DR

This work investigates when preference feedback improves parameter estimation for continuous parametric distributions. It shows that stochastic preferences yield lower asymptotic variance than sample-only estimators, and that deterministic preferences enable a true acceleration to an $\mathcal{O}(1/n)$ estimation rate via hard, constraint-driven updates, with a matching lower bound up to constants. The results hold under general, though restrictive, geometric conditions and are validated in Gaussian and Laplace settings with log-probability rewards. The findings have practical implications for preference-based fine-tuning and iterative alignment, suggesting new estimation paradigms that leverage hard preference constraints for faster learning in high-stakes settings.

Abstract

Recent advances in language modeling have underscored the role of preference feedback in enhancing model performance. This paper investigates the conditions under which preference feedback improves parameter estimation in classes of continuous parametric distributions. In our framework, the learner observes pairs of samples from an unknown distribution along with their relative preferences depending on the same unknown parameter. We show that preference-based M-estimators achieve a better asymptotic variance than sample-only M-estimators, further improved by deterministic preferences. Leveraging the hard constraints revealed by deterministic preferences, we propose an estimator achieving an estimation error scaling of $\mathcal{O}(1/n)$ -- a significant improvement over the $Θ(1/\sqrt{n})$ rate attainable with samples alone. Next, we establish a lower bound that matches this accelerated rate; up to dimension and problem-dependent constants. While the assumptions underpinning our analysis are restrictive, they are satisfied by notable cases such as Gaussian or Laplace distributions for preferences based on the log-probability reward.

Learning Parametric Distributions from Samples and Preferences

TL;DR

This work investigates when preference feedback improves parameter estimation for continuous parametric distributions. It shows that stochastic preferences yield lower asymptotic variance than sample-only estimators, and that deterministic preferences enable a true acceleration to an estimation rate via hard, constraint-driven updates, with a matching lower bound up to constants. The results hold under general, though restrictive, geometric conditions and are validated in Gaussian and Laplace settings with log-probability rewards. The findings have practical implications for preference-based fine-tuning and iterative alignment, suggesting new estimation paradigms that leverage hard preference constraints for faster learning in high-stakes settings.

Abstract

Recent advances in language modeling have underscored the role of preference feedback in enhancing model performance. This paper investigates the conditions under which preference feedback improves parameter estimation in classes of continuous parametric distributions. In our framework, the learner observes pairs of samples from an unknown distribution along with their relative preferences depending on the same unknown parameter. We show that preference-based M-estimators achieve a better asymptotic variance than sample-only M-estimators, further improved by deterministic preferences. Leveraging the hard constraints revealed by deterministic preferences, we propose an estimator achieving an estimation error scaling of -- a significant improvement over the rate attainable with samples alone. Next, we establish a lower bound that matches this accelerated rate; up to dimension and problem-dependent constants. While the assumptions underpinning our analysis are restrictive, they are satisfied by notable cases such as Gaussian or Laplace distributions for preferences based on the log-probability reward.

Paper Structure

This paper contains 103 sections, 9 theorems, 108 equations, 8 figures.

Table of Contents

  1. Introduction
  2. Contributions
  3. Related Work
  4. Learning parametric distributions.
  5. Learning parametric value/preference functions.
  6. Fine-tuning with preference data.
  7. Problem Statement
  8. Parameter estimation.
  9. Preference feedback.
  10. Informative preferences.
  11. Negative examples.
  12. Sample-only MLE
  13. Asymptotic normality.
  14. Regularity conditions.
  15. Preference-based M-estimator
  16. ...and 88 more sections

Key Result

Lemma 3.1

Let $\Delta^{\mathrm{SP}}_{\theta} := \mathbb{E}_{p_{\theta^\star}^{\otimes 2}}[\sigma(\ell_{\theta}) \sigma(-\ell_{\theta}) \nabla_{\theta} \ell_{\theta} \nabla_{\theta} \ell_{\theta}^\mathsf{ T}]$. Then, $\mathcal{I}(q_{\theta^\star, h_{\mathrm{sto}}}) = \mathcal{I}(p_{\theta^\star}^{\otimes 2}

Figures (8)

  • Figure 1: Estimation errors for $\mathcal{N}(\theta^\star,I_d)$ where $\theta^\star \sim \mathcal{U}([1,2]^{d})$ for (a) $d=1$ with $N_{\text{runs}} = 10^3$, and (b) $d=20$ with $N_{\text{runs}} = 10^2$.
  • Figure 2: Estimation errors as a function of $d$ with $\mathcal{N}(\theta^\star,I_d)$ where $\theta^\star \sim \mathcal{U}([1,2]^{d})$, for $n=10^4$ and $N_{\text{runs}} = 10^3$.
  • Figure 3: Approximations of $\Delta^{\mathrm{SP}}_{\mathcal{N}(0_d,I_d)}$, $\Delta^{\mathrm{SP}_{\mathrm{det}}}_{\mathcal{N}(0_d,I_d)}$ and $R^{\mathrm{SP}_{\mathrm{det}}}_{\mathcal{N}(0_d,I_d)}$ by (a) $\alpha_d I_d$ and (b) associated error for varying $d$. $N_{\text{runs}} = 10^6$.
  • Figure 4: Estimation errors for (a) Lap$(\theta^\star,1)$ where $\theta^\star \sim \mathcal{U}([1,2])$ with $N_{\text{runs}} = 10$ and (b) Rayleigh$(\sqrt{\theta^\star})$ where $\theta^\star \sim \mathcal{U}([1,2])$ with $N_{\text{runs}} = 10^2$.
  • Figure 5: Estimation errors for $\mathcal{N}(\theta^\star,I_d)$ where $\theta^\star \sim \mathcal{U}([1,2]^{d})$ with $N_{\text{runs}} = 10^2$ for (a) $d=1$ and (b) $d=20$.
  • ...and 3 more figures

Theorems & Definitions (16)

  • Lemma 3.1
  • Lemma 3.2
  • Lemma 4.1
  • Theorem 4.3
  • Lemma 4.6
  • Theorem 4.8
  • Lemma 5.1
  • Theorem 5.3
  • proof
  • proof
  • ...and 6 more