Comparing Machine Learning Algorithms by Union-Free Generic Depth
Hannah Blocher, Georg Schollmeyer, Malte Nalenz, Christoph Jansen
TL;DR
This work develops a unified depth-based framework for poset-valued data by introducing the union-free generic (ufg) depth, enabling descriptive analysis of multi-criteria classifier performance across data sets. Grounded in closure operators and formal concept analysis, it defines the ufg depth and its empirical counterpart, along with principled bounds and computation strategies. The authors demonstrate the method through two classifier-comparison studies (UCI and OpenML), showing how depth-based centrality and outlier detection provide novel insights beyond traditional benchmarking. The approach supports multidimensional performance evaluation, offers practical implementation guidance, and opens avenues for inference and uncertainty quantification in poset-valued analyses.
Abstract
We propose a framework for descriptively analyzing sets of partial orders based on the concept of depth functions. Despite intensive studies in linear and metric spaces, there is very little discussion on depth functions for non-standard data types such as partial orders. We introduce an adaptation of the well-known simplicial depth to the set of all partial orders, the union-free generic (ufg) depth. Moreover, we utilize our ufg depth for a comparison of machine learning algorithms based on multidimensional performance measures. Concretely, we provide two examples of classifier comparisons on samples of standard benchmark data sets. Our results demonstrate promisingly the wide variety of different analysis approaches based on ufg methods. Furthermore, the examples outline that our approach differs substantially from existing benchmarking approaches, and thus adds a new perspective to the vivid debate on classifier comparison.