Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Human-in-the-loop: Real-time Preference Optimization

Wenbin Wang, Wenjie Xu, Colin N. Jones

TL;DR

This work tackles real-time, human-in-the-loop optimization where user preferences are provided as pairwise comparisons. It introduces an online feedback optimization controller that uses random exploration and binary preference feedback to estimate a descent direction for the latent utility $\tilde{\Phi}(u)=\Phi(h(u),u)$, and proves closed-loop stability and convergence to the unique optimizer $u^*$ under mild assumptions. Theoretical results are backed by Lyapunov-based stability bounds and convergence guarantees, with error terms that vanish as $\delta\to 0$ and $\mu\to 0$. Numerical experiments on a simple quadratic plant and a thermal comfort problem based on PMV/PPD demonstrate practical effectiveness and reveal the impact of plant transients and feedback noise on convergence.

Abstract

Optimization with preference feedback is an active research area with many applications in engineering systems where humans play a central role, such as building control and autonomous vehicles. While most existing studies focus on optimizing a static user utility, few have investigated its closed-loop behavior that accounts for system transients. In this work, we propose an online feedback optimization controller that can optimize user utility using pairwise comparison feedback with both optimality and closed-loop stability guarantees. By adding a random exploration signal, the controller estimates the gradient based on the binary utility comparison feedback between two consecutive time steps. We analyze its closed-loop behavior when interacting with a nonlinear plant and show that, under mild assumptions, the controller converges to the optimal point without inducing instability. Theoretical findings are further validated through numerical experiments.

Human-in-the-loop: Real-time Preference Optimization

TL;DR

This work tackles real-time, human-in-the-loop optimization where user preferences are provided as pairwise comparisons. It introduces an online feedback optimization controller that uses random exploration and binary preference feedback to estimate a descent direction for the latent utility , and proves closed-loop stability and convergence to the unique optimizer under mild assumptions. Theoretical results are backed by Lyapunov-based stability bounds and convergence guarantees, with error terms that vanish as and . Numerical experiments on a simple quadratic plant and a thermal comfort problem based on PMV/PPD demonstrate practical effectiveness and reveal the impact of plant transients and feedback noise on convergence.

Abstract

Optimization with preference feedback is an active research area with many applications in engineering systems where humans play a central role, such as building control and autonomous vehicles. While most existing studies focus on optimizing a static user utility, few have investigated its closed-loop behavior that accounts for system transients. In this work, we propose an online feedback optimization controller that can optimize user utility using pairwise comparison feedback with both optimality and closed-loop stability guarantees. By adding a random exploration signal, the controller estimates the gradient based on the binary utility comparison feedback between two consecutive time steps. We analyze its closed-loop behavior when interacting with a nonlinear plant and show that, under mild assumptions, the controller converges to the optimal point without inducing instability. Theoretical findings are further validated through numerical experiments.

Paper Structure

This paper contains 16 sections, 6 theorems, 22 equations, 2 figures, 1 algorithm.

Table of Contents

  1. Introduction
  2. Problem Formulation and Preliminaries
  3. Notation
  4. Problem Formulation
  5. User model
  6. Online gradient estimation with preference feedback
  7. Performance Analysis
  8. Numerical simulation
  9. A simple example
  10. Thermal comfort optimization
  11. Conclusion
  12. Proof of Lemma \ref{['lemma: lyapunov stable']}
  13. Proof of Lemma \ref{['lemma: p']}
  14. Proof of Lemma \ref{['lemma: unique solution']}
  15. Proof of Lemma \ref{['lemma: bounded error']}
  16. ...and 1 more sections

Key Result

Lemma 1

Let Assumptions ap: map lipschitz-ap: user model hold. $\mathbb{E}[V(x_k,u_k+\delta v_k)]\leq \mu^{k}\mathbb{E}[V(x_0,u_0+\delta v_0)]+\frac{a_1}{1-\mu}(2\delta^2+\eta+(\frac{\eta}{2\delta})^2)$, where $\mu = \frac{2\alpha_2}{\alpha_1}(1-\frac{\alpha_3}{\alpha_2})$, and $a_1 = 4\alpha_2L_{h}^2$.

Figures (2)

  • Figure 1: Quadratic problem.
  • Figure 2: Thermal comfort optimization.

Theorems & Definitions (20)

  • Lemma 1
  • proof
  • Remark 1
  • Lemma 2
  • proof
  • Remark 2
  • Lemma 3
  • proof
  • Lemma 4
  • proof
  • ...and 10 more