Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

MGDA Converges under Generalized Smoothness, Provably

Qi Zhang, Peiyao Xiao, Shaofeng Zou, Kaiyi Ji

TL;DR

This work addresses multi-objective optimization where objective losses satisfy generalized $\ell$-smoothness, a relaxation that captures neural network training dynamics beyond standard $L$-smoothness. It provides a comprehensive convergence theory for MGDA and its stochastic variant under generalized smoothness, focusing on both average and iteration-wise CA distances. The authors introduce a warm-start strategy and an efficient MGDA-FA variant, proving that MGDA can reach an $\epsilon$-accurate Pareto stationary point with $O(\epsilon^{-2})$ (deterministic) or $O(\epsilon^{-4})$ (stochastic) samples for the average CA distance, and tighter per-iteration CA guarantees at the cost of higher sample complexity ($O(\epsilon^{-11})$ deterministic, $O(\epsilon^{-17})$ stochastic). Empirical results on Cityscapes and NYU-v2 validate improved gradient conflict handling and show MGDA-FA’s speed advantage, supporting practical applicability to real-world multi-task settings and informing extensions to related MOO algorithms under generalized smoothness.

Abstract

Multi-objective optimization (MOO) is receiving more attention in various fields such as multi-task learning. Recent works provide some effective algorithms with theoretical analysis but they are limited by the standard $L$-smooth or bounded-gradient assumptions, which typically do not hold for neural networks, such as Long short-term memory (LSTM) models and Transformers. In this paper, we study a more general and realistic class of generalized $\ell$-smooth loss functions, where $\ell$ is a general non-decreasing function of gradient norm. We revisit and analyze the fundamental multiple gradient descent algorithm (MGDA) and its stochastic version with double sampling for solving the generalized $\ell$-smooth MOO problems, which approximate the conflict-avoidant (CA) direction that maximizes the minimum improvement among objectives. We provide a comprehensive convergence analysis of these algorithms and show that they converge to an $ε$-accurate Pareto stationary point with a guaranteed $ε$-level average CA distance (i.e., the gap between the updating direction and the CA direction) over all iterations, where totally $\mathcal{O}(ε^{-2})$ and $\mathcal{O}(ε^{-4})$ samples are needed for deterministic and stochastic settings, respectively. We prove that they can also guarantee a tighter $ε$-level CA distance in each iteration using more samples. Moreover, we analyze an efficient variant of MGDA named MGDA-FA using only $\mathcal{O}(1)$ time and space, while achieving the same performance guarantee as MGDA.

MGDA Converges under Generalized Smoothness, Provably

TL;DR

This work addresses multi-objective optimization where objective losses satisfy generalized -smoothness, a relaxation that captures neural network training dynamics beyond standard -smoothness. It provides a comprehensive convergence theory for MGDA and its stochastic variant under generalized smoothness, focusing on both average and iteration-wise CA distances. The authors introduce a warm-start strategy and an efficient MGDA-FA variant, proving that MGDA can reach an -accurate Pareto stationary point with (deterministic) or (stochastic) samples for the average CA distance, and tighter per-iteration CA guarantees at the cost of higher sample complexity ( deterministic, stochastic). Empirical results on Cityscapes and NYU-v2 validate improved gradient conflict handling and show MGDA-FA’s speed advantage, supporting practical applicability to real-world multi-task settings and informing extensions to related MOO algorithms under generalized smoothness.

Abstract

Multi-objective optimization (MOO) is receiving more attention in various fields such as multi-task learning. Recent works provide some effective algorithms with theoretical analysis but they are limited by the standard -smooth or bounded-gradient assumptions, which typically do not hold for neural networks, such as Long short-term memory (LSTM) models and Transformers. In this paper, we study a more general and realistic class of generalized -smooth loss functions, where is a general non-decreasing function of gradient norm. We revisit and analyze the fundamental multiple gradient descent algorithm (MGDA) and its stochastic version with double sampling for solving the generalized -smooth MOO problems, which approximate the conflict-avoidant (CA) direction that maximizes the minimum improvement among objectives. We provide a comprehensive convergence analysis of these algorithms and show that they converge to an -accurate Pareto stationary point with a guaranteed -level average CA distance (i.e., the gap between the updating direction and the CA direction) over all iterations, where totally and samples are needed for deterministic and stochastic settings, respectively. We prove that they can also guarantee a tighter -level CA distance in each iteration using more samples. Moreover, we analyze an efficient variant of MGDA named MGDA-FA using only time and space, while achieving the same performance guarantee as MGDA.
Paper Structure (37 sections, 18 theorems, 109 equations, 4 figures, 5 tables, 4 algorithms)

This paper contains 37 sections, 18 theorems, 109 equations, 4 figures, 5 tables, 4 algorithms.

Table of Contents

  1. Introduction
  2. Our Contributions
  3. Related Works
  4. Preliminaries
  5. Generalized smoothness
  6. Pareto concepts in multi-objective optimization (MOO)
  7. Existing MOO Algorithms
  8. Conflict-avoidant (CA) direction and CA distance
  9. MGDA Algorithms under Generalized Smoothness
  10. MGDA with and without Warm Start
  11. Stochastic MGDA with Double Sampling
  12. MGDA with Fast Approximation
  13. Convergence Analysis under Average CA distance
  14. Deterministic setting
  15. Stochastic setting
  16. ...and 22 more sections

Key Result

Theorem 1

Let Assumptions ass:diffandlowerbound and ass:lsmooth hold. Set $\beta= \mathcal{O}(\frac{1}{M^2}), \alpha= \mathcal{O}(\frac{1}{M^2}+\frac{1}{M\ell(M+1)}), T= \max(\Theta (\frac{1}{\alpha \epsilon^2}), \Theta\left(\frac{1}{\beta \epsilon^2}\right))$ and $\rho= \mathcal{O}(\epsilon^2)$. We then have

Figures (4)

  • Figure 1: Local smoothness constant vs. gradient norm for training SegNet on CityScapes dataset of Task 1.
  • Figure 2: Multi-task learning on Cityscapes dataset.
  • Figure 3: Local smoothness constant vs. Gradient norm on training SegNet on CityScapes dataset of each task. Task 1 on the left and Task 2 on the right.
  • Figure 4: Multi-task learning on Cityscapes dataset.

Theorems & Definitions (33)

  • Definition 1
  • Definition 2
  • Definition 3
  • Definition 4
  • Theorem 1
  • Corollary 1
  • Theorem 2
  • Lemma 1
  • Theorem 3
  • Theorem 4
  • ...and 23 more