Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Two Criteria for Performance Analysis of Optimization Algorithms

Yunpeng Jing, HaiLin Liu, Qunfeng Liu

TL;DR

Two criteria are introduced to ensure that performance analysis is unaffected by irrelevant factors, including the isomorphism criterion, which asserts that performance evaluation should remain unaffected by the modeling approach, and the IIA criterion, stating that comparisons between two algorithms should not be influenced by irrelevant third-party algorithms.

Abstract

Performance analysis is crucial in optimization research, especially when addressing black-box problems through nature-inspired algorithms. Current practices often rely heavily on statistical methods, which can lead to various logical paradoxes. To address this challenge, this paper introduces two criteria to ensure that performance analysis is unaffected by irrelevant factors. The first is the isomorphism criterion, which asserts that performance evaluation should remain unaffected by the modeling approach. The second is the IIA criterion,stating that comparisons between two algorithms should not be influenced by irrelevant third-party algorithms. Additionally, we conduct a comprehensive examination of the underlying causes of these paradoxes, identify conditions for checking the criteria, and propose ideas to tackle these issues. The criteria presented offer a framework for researchers to critically assess the performance metrics or ranking methods, ultimately aiming to enhance the rigor of evaluation metrics and ranking methods.

Two Criteria for Performance Analysis of Optimization Algorithms

TL;DR

Two criteria are introduced to ensure that performance analysis is unaffected by irrelevant factors, including the isomorphism criterion, which asserts that performance evaluation should remain unaffected by the modeling approach, and the IIA criterion, stating that comparisons between two algorithms should not be influenced by irrelevant third-party algorithms.

Abstract

Performance analysis is crucial in optimization research, especially when addressing black-box problems through nature-inspired algorithms. Current practices often rely heavily on statistical methods, which can lead to various logical paradoxes. To address this challenge, this paper introduces two criteria to ensure that performance analysis is unaffected by irrelevant factors. The first is the isomorphism criterion, which asserts that performance evaluation should remain unaffected by the modeling approach. The second is the IIA criterion,stating that comparisons between two algorithms should not be influenced by irrelevant third-party algorithms. Additionally, we conduct a comprehensive examination of the underlying causes of these paradoxes, identify conditions for checking the criteria, and propose ideas to tackle these issues. The criteria presented offer a framework for researchers to critically assess the performance metrics or ranking methods, ultimately aiming to enhance the rigor of evaluation metrics and ranking methods.

Paper Structure

This paper contains 16 sections, 5 theorems, 12 equations, 4 figures, 2 tables.

Table of Contents

  1. Introduction
  2. The isomorphism criterion
  3. The motivation of isomorphism criterion
  4. The model of isomorphism
  5. The isomorphism criterion
  6. Isomorphism analysis for metrics in practice
  7. Why does isomorphism matter?
  8. A necessary condition for isomorphism metrics
  9. Isomorphism normalization: a solution idea
  10. The IIA criterion
  11. The motivations of IIA criterion
  12. The IIA criterion
  13. The examples of IIA paradox
  14. An analysis of the cause of the IIA paradox
  15. A response idea to the IIA paradox
  16. ...and 1 more sections

Key Result

Proposition 1

$f\sim g$ if and only if $\forall x_1, x_2$, $sign(f(x_1)-f(x_2))=sign(g(x_1)-g(x_2))$, where $sign$ is the signum function.

Figures (4)

  • Figure 1: Real-world problem and functional problems. A real-world search process produces the same actual results; however, when the problem is modeled using different functions, the function values may differ. Despite this, because the real-world results are the same, the performance evaluations should remain invariant.
  • Figure 2: IIA Paradox and Simpson's Paradox. Simpson's paradox emphasizes that comparison results are influenced by the set of benchmark functions, and IIA paradox symmetrically emphasizes that comparison results are influenced by the set of algorithms.
  • Figure 3: A schematic diagram illustrating the IIA paraodx. The irrelevant algorithm $C$ unexpectedly influenced the comparison result of $A$ and $B$.
  • Figure 4: The erasure effect generated by using relative rankings. Algorithms positioned adjacently, regardless of the magnitude of their differences in raw data, have their gaps set to 1 after ranking. On the contrary, adding sufficient algorithms allows the performance differences between adjacent algorithms to become evident in their scores.

Theorems & Definitions (14)

  • Definition 1: isomorphism criterion
  • Definition 2: isomorphism
  • Proposition 1: comparison invariance
  • Proposition 2: monotonic invariance
  • Definition 3: linear isomorphism
  • Definition 4: isomorphism set
  • Definition 5: isomorphism criterion
  • Lemma 1
  • Proposition 3
  • Definition 6
  • ...and 4 more