Non-Stationary Functional Bilevel Optimization

TL;DR

This work extends functional bilevel optimization to online, non-stationary environments by introducing SmoothFBO, which uses time-smoothed hypergradients to stabilize updates and achieve sublinear regret. The approach generalizes FBO to time-varying data and provides theoretical guarantees, including bias/variance bounds and regret analyses, alongside practical validation in non-stationary regression and model-based reinforcement learning. Key contributions include a formal non-stationary FBO framework, a time-smoothed hypergradient estimator with variance reduction, and empirical demonstrations of robust adaptation to drift in both synthetic and RL tasks. The results suggest SmoothFBO as a principled, scalable foundation for online bilevel optimization in function spaces, with potential for broader variance-reduction techniques in dynamic settings.

Abstract

Functional bilevel optimization (FBO) provides a powerful framework for hierarchical learning in function spaces, yet current methods are limited to static offline settings and perform suboptimally in online, non-stationary scenarios. We propose SmoothFBO, the first algorithm for non-stationary FBO with both theoretical guarantees and practical scalability. SmoothFBO introduces a time-smoothed stochastic hypergradient estimator that reduces variance through a window parameter, enabling stable outer-loop updates with sublinear regret. Importantly, the classical parametric bilevel case is a special reduction of our framework, making SmoothFBO a natural extension to online, non-stationary settings. Empirically, SmoothFBO consistently outperforms existing FBO methods in non-stationary hyperparameter optimization and model-based reinforcement learning, demonstrating its practical effectiveness. Together, these results establish SmoothFBO as a general, theoretically grounded, and practically viable foundation for bilevel optimization in online, non-stationary scenarios.

Figures (8)

• Figure 1: Bilevel local regret (BLR$_\omega$) vs. rounds (Fig. 1). SmoothFBO achieves sublinear regret, consistent with our theorem. The zoomed-in component highlights the sublinear trend, while increasing the window $w$ ($5\!\to\!500$) further reduces regret.
• Figure 2: Parameter drift in the underlying DGP (Fig. 2). The weights $(W_t, b_t)$ evolve nonstationarily, driving the outer-loop adaptation challenge.
• Figure 3: The changing pole angle throughout training. In the non-stationary CartPole experiment, the target pole angle shifts gradually over training steps, forcing the agent to adapt its learned dynamics and policy.
• Figure 4: Cumulative reward for the non-stationary CartPole evaluation environment over 1 million environment steps. Each curve represents the mean cumulative episode reward across 10 random seeds, with shaded regions indicating 95% confidence intervals. Left: The FBO method in stationary and non-stationary environments compared to SmoothFBO. Right: Comparison with baseline methods where SmoothFBO matches their performance in adapting to the non-stationary dynamics.
• Figure 5: Effect of smoothing on regret and hypergradient variance. (Left) Cumulative bilevel local regret (BLR$_\omega$) on the sinusoidal-drift task, showing sublinear regret for SmoothFBO; curves are averaged over seeds. (Right) Variance of the hypergradient as a function of the smoothing window $w$; increasing $w$ reduces variance. These results demonstrate that temporal smoothing stabilizes bilevel optimization and improves bilevel local regret. Shaded regions indicate $95\%$ confidence intervals across seeds.

Theorems & Definitions (28)

• Definition 3.1: Stochastic Hypergradient Oracle
• Lemma 3.2: Time-Smoothed Hypergradient Estimator
• proof
• Definition 3.3: Bilevel Local Regret
• Theorem 3.4
• Corollary 3.5
• Lemma 4.1: Time-Smoothed Hypergradient Estimator
• proof
• Lemma 4.2: Expected Squared Error of Time-Smoothed Hypergradient Estimator
• Theorem 4.3