Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

PMGDA: A Preference-based Multiple Gradient Descent Algorithm

Xiaoyuan Zhang, Xi Lin, Qingfu Zhang

TL;DR

This work tackles the challenge of finding Pareto solutions in large-scale multi-objective optimization that align with a decision-maker's preferences. It introduces PMGDA, a predict-and-correct framework built on MGDA that optimizes both the objective vector and a user-specified constraint $h(\bm{\theta})$, enabling exact Pareto and ROI-focused solutions. The method leverages dual formulations for efficient prediction and a small dual LP for correction, achieving scalability to neural networks with thousands of parameters and adapting to MORL tasks. Empirical results on synthetic benchmarks, fairness multitask learning, and MORL demonstrate superior precision, ROI satisfaction, and faster convergence compared with prior approaches, with code to be released. The framework thus offers a practical, flexible pathway for preference-guided multi-objective learning in real-world, large-scale settings.

Abstract

It is desirable in many multi-objective machine learning applications, such as multi-task learning with conflicting objectives and multi-objective reinforcement learning, to find a Pareto solution that can match a given preference of a decision maker. These problems are often large-scale with available gradient information but cannot be handled very well by the existing algorithms. To tackle this critical issue, this paper proposes a novel predict-and-correct framework for locating a Pareto solution that fits the preference of a decision maker. In the proposed framework, a constraint function is introduced in the search progress to align the solution with a user-specific preference, which can be optimized simultaneously with multiple objective functions. Experimental results show that our proposed method can efficiently find a particular Pareto solution under the demand of a decision maker for standard multiobjective benchmark, multi-task learning, and multi-objective reinforcement learning problems with more than thousands of decision variables. Code is available at: https://github.com/xzhang2523/pmgda. Our code is current provided in the pgmda.rar attached file and will be open-sourced after publication.}

PMGDA: A Preference-based Multiple Gradient Descent Algorithm

TL;DR

This work tackles the challenge of finding Pareto solutions in large-scale multi-objective optimization that align with a decision-maker's preferences. It introduces PMGDA, a predict-and-correct framework built on MGDA that optimizes both the objective vector and a user-specified constraint , enabling exact Pareto and ROI-focused solutions. The method leverages dual formulations for efficient prediction and a small dual LP for correction, achieving scalability to neural networks with thousands of parameters and adapting to MORL tasks. Empirical results on synthetic benchmarks, fairness multitask learning, and MORL demonstrate superior precision, ROI satisfaction, and faster convergence compared with prior approaches, with code to be released. The framework thus offers a practical, flexible pathway for preference-guided multi-objective learning in real-world, large-scale settings.

Abstract

It is desirable in many multi-objective machine learning applications, such as multi-task learning with conflicting objectives and multi-objective reinforcement learning, to find a Pareto solution that can match a given preference of a decision maker. These problems are often large-scale with available gradient information but cannot be handled very well by the existing algorithms. To tackle this critical issue, this paper proposes a novel predict-and-correct framework for locating a Pareto solution that fits the preference of a decision maker. In the proposed framework, a constraint function is introduced in the search progress to align the solution with a user-specific preference, which can be optimized simultaneously with multiple objective functions. Experimental results show that our proposed method can efficiently find a particular Pareto solution under the demand of a decision maker for standard multiobjective benchmark, multi-task learning, and multi-objective reinforcement learning problems with more than thousands of decision variables. Code is available at: https://github.com/xzhang2523/pmgda. Our code is current provided in the pgmda.rar attached file and will be open-sourced after publication.}
Paper Structure (22 sections, 1 theorem, 21 equations, 12 figures, 5 tables, 1 algorithm)

This paper contains 22 sections, 1 theorem, 21 equations, 12 figures, 5 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Background
  3. Notations
  4. Basic MOO Definitions
  5. The Multiple Gradient Descent Algorithm (MGDA)
  6. The Preference-based Multi-Gradient Descent Algorithm (PMGDA)
  7. Preference Functions
  8. The exact preference function
  9. The Region of Interest (ROI) preference function
  10. The Predict-and-Correct Framework
  11. An Efficient Prediction Algorithm
  12. An Efficient Correction Algorithm
  13. Reuse of Gradients
  14. Algorithm and its Complexity Analysis
  15. Multi-objective Reinforcement Learning (MORL) Task
  16. ...and 7 more sections

Key Result

Lemma 1

When ${\bm{\theta}} _{k}$ is not Pareto stationary, $0 < \left\Vert {\bm{v}} \right\Vert \leq 1$.

Figures (12)

  • Figure 1: Two particular cases of preference-based MOO. The left figure: the exact Pareto solution aligns with a given preference vector. The right figure: ROI Pareto solutions satisfy the ROI constraint.
  • Figure 2: The optimization trajectory of $\sigma=0.85, 0.95, 0.99$. A small setting of $\sigma$ leads to a smooth optimization trajectory, while a large set of $\sigma$ leads to oscillations and a slow convergence rate. Exact Pareto solutions are found separately in 127, 144, and 305 iterations. Marker $\star$ denotes the final solutions.
  • Figure 4: Learning curves on ZDT1. The proposed method has the best convergence speed and final performance. The convergence curve of EPO fluctuates before convergence. Final solutions of COSMOS are also not exact Pareto solutions.
  • Figure 5: Learning curves on MAF1. Final solutions by COSMOS and EPO are not exact Pareto solutions. On this three-objective problem, the convergence of mTche is slow, taking 500 iterations to converge, while PMGDA only needs 100 iterations to converge.
  • Figure 6: Learning curves on DTLZ2. PMGDA and COSMOS successfully finds all exact Pareto solutions (PMGDA is 3x faster than COSMOS). However, EPO and mTche fail to find all exact Pareto solutions in 500 iterations. The learning curve of EPO is unstable.
  • ...and 7 more figures

Theorems & Definitions (6)

  • Definition 1: Dominance miettinen2012nonlinear(Chap. 2.2)
  • Definition 2: Pareto Optimality miettinen2012nonlinear(Chap. 2.2)
  • Definition 3: Pareto Stationary Solution
  • Definition 4: 'Exact' Pareto Solution
  • Lemma 1
  • proof