Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Model-Agnostic Meta-Policy Optimization via Zeroth-Order Estimation: A Linear Quadratic Regulator Perspective

Yunian Pan, Tao Li, Quanyan Zhu

TL;DR

This work tackles model-agnostic meta-policy optimization for a collection of ergodic LQR tasks with unknown dynamics by learning a meta-initialization $K$ that enables rapid one-step adaptation. It introduces a Hessian-free, zeroth-order meta-gradient method based on Stein's Gaussian smoothing to avoid computing second-order information, enabling scalable meta-learning across similar LTIs. Theoretical contributions include stability and convergence guarantees, with explicit sample-complexity conditions for gradient and meta-gradient estimators, and finite-time bounds. Numerical experiments on five related LQR instances demonstrate cross-task performance gains, validating the approach for model-free policy optimization in linear control settings.

Abstract

Meta-learning has been proposed as a promising machine learning topic in recent years, with important applications to image classification, robotics, computer games, and control systems. In this paper, we study the problem of using meta-learning to deal with uncertainty and heterogeneity in ergodic linear quadratic regulators. We integrate the zeroth-order optimization technique with a typical meta-learning method, proposing an algorithm that omits the estimation of policy Hessian, which applies to tasks of learning a set of heterogeneous but similar linear dynamic systems. The induced meta-objective function inherits important properties of the original cost function when the set of linear dynamic systems are meta-learnable, allowing the algorithm to optimize over a learnable landscape without projection onto the feasible set. We provide stability and convergence guarantees for the exact gradient descent process by analyzing the boundedness and local smoothness of the gradient for the meta-objective, which justify the proposed algorithm with gradient estimation error being small. We provide the sample complexity conditions for these theoretical guarantees, as well as a numerical example at the end to corroborate this perspective.

Model-Agnostic Meta-Policy Optimization via Zeroth-Order Estimation: A Linear Quadratic Regulator Perspective

TL;DR

This work tackles model-agnostic meta-policy optimization for a collection of ergodic LQR tasks with unknown dynamics by learning a meta-initialization that enables rapid one-step adaptation. It introduces a Hessian-free, zeroth-order meta-gradient method based on Stein's Gaussian smoothing to avoid computing second-order information, enabling scalable meta-learning across similar LTIs. Theoretical contributions include stability and convergence guarantees, with explicit sample-complexity conditions for gradient and meta-gradient estimators, and finite-time bounds. Numerical experiments on five related LQR instances demonstrate cross-task performance gains, validating the approach for model-free policy optimization in linear control settings.

Abstract

Meta-learning has been proposed as a promising machine learning topic in recent years, with important applications to image classification, robotics, computer games, and control systems. In this paper, we study the problem of using meta-learning to deal with uncertainty and heterogeneity in ergodic linear quadratic regulators. We integrate the zeroth-order optimization technique with a typical meta-learning method, proposing an algorithm that omits the estimation of policy Hessian, which applies to tasks of learning a set of heterogeneous but similar linear dynamic systems. The induced meta-objective function inherits important properties of the original cost function when the set of linear dynamic systems are meta-learnable, allowing the algorithm to optimize over a learnable landscape without projection onto the feasible set. We provide stability and convergence guarantees for the exact gradient descent process by analyzing the boundedness and local smoothness of the gradient for the meta-objective, which justify the proposed algorithm with gradient estimation error being small. We provide the sample complexity conditions for these theoretical guarantees, as well as a numerical example at the end to corroborate this perspective.

Paper Structure

This paper contains 18 sections, 21 theorems, 114 equations, 1 figure, 3 algorithms.

Table of Contents

  1. INTRODUCTION
  2. RELATED WORK
  3. Policy Optimization (PO)
  4. Model-Agnostic Meta-Learning (MAML)
  5. PROBLEM FORMULATION
  6. Preliminary: Policy Optimization for LQRs
  7. Meta-Policy-Optimization
  8. METHODOLOGY
  9. Zero-th Order Methods
  10. Hessian-Free Meta-Gradient Estimation
  11. GRADIENT DESCENT ANALYSIS
  12. Controlling Estimation Error
  13. Theoretical Guarantee
  14. NUMERICAL RESULTS
  15. CONCLUSIONS
  16. ...and 3 more sections

Key Result

Proposition 1

For any task $\mathcal{T}_i$, the expression for average cost is $J_i(K) = \operatorname{Tr}(P^i_K )$, and the expression of $\nabla J_i(K)$ is where $\Sigma^i_{K}$ satisfies xgramian, $E^i_K$ is defined to be and $P^i_K$ is the unique positive definite solution to the Lyapunov equation. The Hessian operator $\nabla J_i(K)$ acting on some $X \in \mathop{\mathrm{\mathbb{R}}}\limits^{ k \times d}

Figures (1)

  • Figure 1: The plot shows three curves encapsulating the changing of average performance during gradient descent, each corresponds to a particular dimension setting of state and action space, (green: $d = 20, k = 10$, orange: $d = 2, k = 2$, blue: $d= 1, k =1$.) constant learning rates $\alpha = 1e-3$, $\eta = 1e-5$ for orange and blue cases and $\alpha = 1e-5$, $\eta = 1e-7$ for green curve, numbers of meta and inner perturbation $M= 100$, gradient smooth parameter $r = 0.05$, roll out length $\ell = 50$.

Theorems & Definitions (31)

  • Proposition 1: Policy Gradient for LQR fazel2018globalDBLP:journals/corr/abs-1907-06246bu2019lqrlensordermethods
  • Definition 1: MAML-stablizing musavi2023convergence
  • Definition 2: Stabilizing sub-level set toso2024meta
  • lemma 1: Uniform bounds toso2024meta
  • lemma 2: Perturbation Analysis toso2024metamusavi2023convergence
  • lemma 3: Gradient Domination fazel2018globalyang19pg-lqr
  • lemma 4
  • lemma 5: Perturbation analysis of $\nabla \mathcal{L}(K)$
  • lemma 6: Gradient Estimation
  • lemma 7: Meta-gradient Estimation
  • ...and 21 more