Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Unlearning Works Better Than You Think: Local Reinforcement-Based Selection of Auxiliary Objectives

Abderrahim Bendahi, Adrien Fradin, Matthieu Lerasle

TL;DR

This work tackles non-monotonic single-objective optimization by augmenting an evolutionary algorithm with reinforcement learning to select auxiliary objectives, while introducing an unlearning mechanism that discards no-longer-beneficial helpers. The Local Reinforcement-Based Selection of Auxiliary Objectives (LRSAO) uses a local reward structure and a plateau-aware penalty to adaptively switch or discard objectives like LeftBridge and RightBridge as the Jump$_{\ell}$ landscape changes, without requiring restarts. The authors prove a tight runtime bound of $\mathbb{E}(T)=\Theta\left(\frac{n^2}{\ell^2} + n\log n\right)$, improving over the previous $O\left(\frac{n^2\log n}{\ell}\right)$, and corroborate the analysis with experiments up to $n=10^4$. The results demonstrate that locality-driven learning and unlearning can significantly enhance the efficiency of RL-guided objective selection in complex optimization landscapes, with practical implications for adaptive heuristic design in evolutionary computation.

Abstract

We introduce Local Reinforcement-Based Selection of Auxiliary Objectives (LRSAO), a novel approach that selects auxiliary objectives using reinforcement learning (RL) to support the optimization process of an evolutionary algorithm (EA) as in EA+RL framework and furthermore incorporates the ability to unlearn previously used objectives. By modifying the reward mechanism to penalize moves that do no increase the fitness value and relying on the local auxiliary objectives, LRSAO dynamically adapts its selection strategy to optimize performance according to the landscape and unlearn previous objectives when necessary. We analyze and evaluate LRSAO on the black-box complexity version of the non-monotonic Jump function, with gap parameter $\ell$, where each auxiliary objective is beneficial at specific stages of optimization. The Jump function is hard to optimize for evolutionary-based algorithms and the best-known complexity for reinforcement-based selection on Jump was $O(n^2 \log(n) / \ell)$. Our approach improves over this result to achieve a complexity of $Θ(n^2 / \ell^2 + n \log(n))$ resulting in a significant improvement, which demonstrates the efficiency and adaptability of LRSAO, highlighting its potential to outperform traditional methods in complex optimization scenarios.

Unlearning Works Better Than You Think: Local Reinforcement-Based Selection of Auxiliary Objectives

TL;DR

This work tackles non-monotonic single-objective optimization by augmenting an evolutionary algorithm with reinforcement learning to select auxiliary objectives, while introducing an unlearning mechanism that discards no-longer-beneficial helpers. The Local Reinforcement-Based Selection of Auxiliary Objectives (LRSAO) uses a local reward structure and a plateau-aware penalty to adaptively switch or discard objectives like LeftBridge and RightBridge as the Jump landscape changes, without requiring restarts. The authors prove a tight runtime bound of , improving over the previous , and corroborate the analysis with experiments up to . The results demonstrate that locality-driven learning and unlearning can significantly enhance the efficiency of RL-guided objective selection in complex optimization landscapes, with practical implications for adaptive heuristic design in evolutionary computation.

Abstract

We introduce Local Reinforcement-Based Selection of Auxiliary Objectives (LRSAO), a novel approach that selects auxiliary objectives using reinforcement learning (RL) to support the optimization process of an evolutionary algorithm (EA) as in EA+RL framework and furthermore incorporates the ability to unlearn previously used objectives. By modifying the reward mechanism to penalize moves that do no increase the fitness value and relying on the local auxiliary objectives, LRSAO dynamically adapts its selection strategy to optimize performance according to the landscape and unlearn previous objectives when necessary. We analyze and evaluate LRSAO on the black-box complexity version of the non-monotonic Jump function, with gap parameter , where each auxiliary objective is beneficial at specific stages of optimization. The Jump function is hard to optimize for evolutionary-based algorithms and the best-known complexity for reinforcement-based selection on Jump was . Our approach improves over this result to achieve a complexity of resulting in a significant improvement, which demonstrates the efficiency and adaptability of LRSAO, highlighting its potential to outperform traditional methods in complex optimization scenarios.

Paper Structure

This paper contains 23 sections, 23 theorems, 20 equations, 4 figures, 1 table, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Works
  3. Problem Statement
  4. The Auxiliary and Target Objectives
  5. The LRSAO Algorithm
  6. Notation and the Main Assumption
  7. Notation
  8. The Main Assumption
  9. Runtime Analysis of LRSAO
  10. Main Result
  11. A Global Upper Bound and Some Properties
  12. Key Lemma for the Plateaus
  13. The First Phase: Learning LeftBridge
  14. The Second Phase: Climbing the Slope
  15. The Third Phase: Unlearn LeftBridge
  16. ...and 8 more sections

Key Result

Theorem 1

The LRSAO algorithm optimizes the black-box Jump$_{\ell}$ function with an average runtime of

Figures (4)

  • Figure 1: The three objectives.
  • Figure 4: Total average runtime with $95\%$ confident intervals.
  • Figure 5: Total average runtime, LRSAO (orange) vs. EA+RL (blue).
  • Figure 6: Total average runtime of LRSAO for various $\ell$.

Theorems & Definitions (24)

  • Theorem 1: Total Average Runtime
  • Lemma 2
  • Lemma 3: $Q$-Table and a Local Maximum
  • Lemma 4
  • Lemma 5
  • Lemma 6: Few Mistakes Lemma
  • Theorem 7: Runtime of the First Phase
  • Lemma 8: Bounds on the $Q$-Table
  • Lemma 9
  • Lemma 10: Few Visits Lemma
  • ...and 14 more