Common pitfalls to avoid while using multiobjective optimization in machine learning

TL;DR

This document serves as a comprehensive guide to the SIAM LaTeX style, detailing how to prepare manuscripts for SIAM journals. It covers class options, front matter, cross-referencing, math and theorem environments, tables, figures, algorithms, and sections, as well as the use of supplementary materials, templates, and enhanced bibliographic fields. The text emphasizes practical workflow aspects such as hyperlinking, PDF bookmarks, and template-driven structures, while outlining major changes and customization options for authors. By consolidating formatting rules, template usage, and BibTeX enhancements, it enables consistent, publication-ready submissions and smoother integration with SIAM's editorial pipeline. The practical impact lies in streamlined manuscript preparation, improved traceability of references and software, and robust support for supplementary materials and modern citation metadata.

Abstract

Recently, there has been an increasing interest in the application of multiobjective optimization (MOO) in machine learning (ML). This interest is driven by the numerous real-life situations where multiple objectives must be optimized simultaneously. A key aspect of MOO is the existence of a Pareto set, rather than a single optimal solution, which represents the optimal trade-offs between different objectives. Despite its potential, there is a noticeable lack of satisfactory literature serving as an entry-level guide for ML practitioners aiming to apply MOO effectively. In this paper, our goal is to provide such a resource and highlight pitfalls to avoid. We begin by establishing the groundwork for MOO, focusing on well-known approaches such as the weighted sum (WS) method, alongside more advanced techniques like the multiobjective gradient descent algorithm (MGDA). We critically review existing studies across various ML fields where MOO has been applied and identify challenges that can lead to incorrect interpretations. One of these fields is physics informed neural networks (PINNs), which we use as a guiding example to carefully construct experiments illustrating these pitfalls. By comparing WS and MGDA with one of the most common evolutionary algorithms, NSGA-II, we demonstrate that difficulties can arise regardless of the specific MOO method used. We emphasize the importance of understanding the specific problem, the objective space, and the selected MOO method, while also noting that neglecting factors such as convergence criteria can result in misleading experiments.

Figures (2)

• Figure 1: Example figure using external image files.
• Figure 2: Example PGFPLOTS figure.

Theorems & Definitions (5)

• Theorem 6.1: Mean Value Theorem
• Corollary 6.2
• Proof 1
• Claim 6.3
• Proof 2: Proof of main theorem