Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Improving Bayesian Optimization for Portfolio Management with an Adaptive Scheduling

Zinuo You, John Cartlidge, Karen Elliott, Menghan Ge, Daniel Gold

TL;DR

The paper addresses the instability and inefficiency of optimizing expensive, opaque portfolio models under tight evaluation budgets. It proposes TPE-AS, a Bayesian optimization framework that couples a Tree-structured Parzen Estimator surrogate with an adaptive weighted Lagrangian objective and clipped importance sampling to jointly maximize expected performance and minimize observation variance. The approach reframes the search as a bi-objective problem and uses a time-varying penalty to shift focus from exploration to stability, yielding smoother, more reliable optimization trajectories. Empirical results across four backtest settings and three black-box portfolio models show improved Sharpe ratios, reduced variance, and faster runtimes, demonstrating practical benefits for stable, sample-efficient tuning in finance under budget constraints.

Abstract

Existing black-box portfolio management systems are prevalent in the financial industry due to commercial and safety constraints, though their performance can fluctuate dramatically with changing market regimes. Evaluating these non-transparent systems is computationally expensive, as fixed budgets limit the number of possible observations. Therefore, achieving stable and sample-efficient optimization for these systems has become a critical challenge. This work presents a novel Bayesian optimization framework (TPE-AS) that improves search stability and efficiency for black-box portfolio models under these limited observation budgets. Standard Bayesian optimization, which solely maximizes expected return, can yield erratic search trajectories and misalign the surrogate model with the true objective, thereby wasting the limited evaluation budget. To mitigate these issues, we propose a weighted Lagrangian estimator that leverages an adaptive schedule and importance sampling. This estimator dynamically balances exploration and exploitation by incorporating both the maximization of model performance and the minimization of the variance of model observations. It guides the search from broad, performance-seeking exploration towards stable and desirable regions as the optimization progresses. Extensive experiments and ablation studies, which establish our proposed method as the primary approach and other configurations as baselines, demonstrate its effectiveness across four backtest settings with three distinct black-box portfolio management models.

Improving Bayesian Optimization for Portfolio Management with an Adaptive Scheduling

TL;DR

The paper addresses the instability and inefficiency of optimizing expensive, opaque portfolio models under tight evaluation budgets. It proposes TPE-AS, a Bayesian optimization framework that couples a Tree-structured Parzen Estimator surrogate with an adaptive weighted Lagrangian objective and clipped importance sampling to jointly maximize expected performance and minimize observation variance. The approach reframes the search as a bi-objective problem and uses a time-varying penalty to shift focus from exploration to stability, yielding smoother, more reliable optimization trajectories. Empirical results across four backtest settings and three black-box portfolio models show improved Sharpe ratios, reduced variance, and faster runtimes, demonstrating practical benefits for stable, sample-efficient tuning in finance under budget constraints.

Abstract

Existing black-box portfolio management systems are prevalent in the financial industry due to commercial and safety constraints, though their performance can fluctuate dramatically with changing market regimes. Evaluating these non-transparent systems is computationally expensive, as fixed budgets limit the number of possible observations. Therefore, achieving stable and sample-efficient optimization for these systems has become a critical challenge. This work presents a novel Bayesian optimization framework (TPE-AS) that improves search stability and efficiency for black-box portfolio models under these limited observation budgets. Standard Bayesian optimization, which solely maximizes expected return, can yield erratic search trajectories and misalign the surrogate model with the true objective, thereby wasting the limited evaluation budget. To mitigate these issues, we propose a weighted Lagrangian estimator that leverages an adaptive schedule and importance sampling. This estimator dynamically balances exploration and exploitation by incorporating both the maximization of model performance and the minimization of the variance of model observations. It guides the search from broad, performance-seeking exploration towards stable and desirable regions as the optimization progresses. Extensive experiments and ablation studies, which establish our proposed method as the primary approach and other configurations as baselines, demonstrate its effectiveness across four backtest settings with three distinct black-box portfolio management models.

Paper Structure

This paper contains 15 sections, 7 equations, 1 figure, 3 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Portfolio Optimization Models
  4. Bayesian Optimization
  5. Problem Formulation
  6. Methodology
  7. The TPE framework
  8. Adaptive Weighted Lagrangian Estimator
  9. Dual Objectives
  10. Importance Sampling
  11. Experiments
  12. Experimental Setting
  13. Performance Evaluation
  14. Ablation Study
  15. Conclusions

Figures (1)

  • Figure 1: The x-axis shows 500 optimization steps. Blue dots represent portfolio model performance; green dots represent the corresponding objective score for each optimization step.