Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

On Admissible Rank-based Input Normalization Operators

Taeyun Kim

TL;DR

We address the stability and invariance of rank-based input normalization by imposing three axioms: rank-level invariance under strictly increasing feature-wise transforms (C1), batch independence (C2), and monotone Lipschitz-stable scalarization (C3). The main result proves that any admissible operator must factor through a feature-wise rank representation and a monotone Lipschitz scalarization, i.e., $Q(x)=\Phi\bigl(s(r(x))\bigr)$; thus differentiable sorting methods based on value gaps or batch interactions lie outside this class. The paper introduces a minimal realization called QNorm and an explicit rank-based representation layer, plus empirical validation across operator stability, model robustness, and real-world datasets demonstrating nontrivial, stable behavior under monotone transformations and batch changes. This work delineates the design space for rank-based input normalization, offering a principled framework to analyze, compare, and design admissible normalization operators with provable invariance and stability properties.

Abstract

Rank-based input normalization is a workhorse of modern machine learning, prized for its robustness to scale, monotone transformations, and batch-to-batch variation. In many real systems, the ordering of feature values matters far more than their raw magnitudes - yet the structural conditions that a rank-based normalization operator must satisfy to remain stable under these invariances have never been formally pinned down. We show that widely used differentiable sorting and ranking operators fundamentally fail these criteria. Because they rely on value gaps and batch-level pairwise interactions, they are intrinsically unstable under strictly monotone transformations, shifts in mini-batch composition, and even tiny input perturbations. Crucially, these failures stem from the operators' structural design, not from incidental implementation choices. To address this, we propose three axioms that formalize the minimal invariance and stability properties required of rank-based input normalization. We prove that any operator satisfying these axioms must factor into (i) a feature-wise rank representation and (ii) a scalarization map that is both monotone and Lipschitz-continuous. We then construct a minimal operator that meets these criteria and empirically show that the resulting constraints are non-trivial in realistic setups. Together, our results sharply delineate the design space of valid rank-based normalization operators and formally separate them from existing continuous-relaxation-based sorting methods.

On Admissible Rank-based Input Normalization Operators

TL;DR

We address the stability and invariance of rank-based input normalization by imposing three axioms: rank-level invariance under strictly increasing feature-wise transforms (C1), batch independence (C2), and monotone Lipschitz-stable scalarization (C3). The main result proves that any admissible operator must factor through a feature-wise rank representation and a monotone Lipschitz scalarization, i.e., ; thus differentiable sorting methods based on value gaps or batch interactions lie outside this class. The paper introduces a minimal realization called QNorm and an explicit rank-based representation layer, plus empirical validation across operator stability, model robustness, and real-world datasets demonstrating nontrivial, stable behavior under monotone transformations and batch changes. This work delineates the design space for rank-based input normalization, offering a principled framework to analyze, compare, and design admissible normalization operators with provable invariance and stability properties.

Abstract

Rank-based input normalization is a workhorse of modern machine learning, prized for its robustness to scale, monotone transformations, and batch-to-batch variation. In many real systems, the ordering of feature values matters far more than their raw magnitudes - yet the structural conditions that a rank-based normalization operator must satisfy to remain stable under these invariances have never been formally pinned down. We show that widely used differentiable sorting and ranking operators fundamentally fail these criteria. Because they rely on value gaps and batch-level pairwise interactions, they are intrinsically unstable under strictly monotone transformations, shifts in mini-batch composition, and even tiny input perturbations. Crucially, these failures stem from the operators' structural design, not from incidental implementation choices. To address this, we propose three axioms that formalize the minimal invariance and stability properties required of rank-based input normalization. We prove that any operator satisfying these axioms must factor into (i) a feature-wise rank representation and (ii) a scalarization map that is both monotone and Lipschitz-continuous. We then construct a minimal operator that meets these criteria and empirically show that the resulting constraints are non-trivial in realistic setups. Together, our results sharply delineate the design space of valid rank-based normalization operators and formally separate them from existing continuous-relaxation-based sorting methods.
Paper Structure (68 sections, 9 theorems, 97 equations, 2 figures, 2 tables)

This paper contains 68 sections, 9 theorems, 97 equations, 2 figures, 2 tables.

Table of Contents

  1. Keywords:
  2. Introduction
  3. Background and Related Work
  4. Differentiable Sorting and Continuous Relaxations
  5. Learning to Rank and Rank-Based Objectives
  6. Normalization, Invariance, and Stability
  7. Structural Conditions
  8. Rank-Equivalence and Monotone Transformations
  9. Admissibility Conditions
  10. (C1) Feature-wise rank-level monotone invariance.
  11. (C2) Batch independence.
  12. (C3) Monotone and Lipschitz-stable scalarization.
  13. Problem statement
  14. Structural Incompatibility with Admissible Conditions
  15. Value-gap Sensitivity
  16. ...and 53 more sections

Key Result

Theorem 1

Let $Q : \mathbb{R}^d \rightarrow [0,1]$ be an operator that satisfies conditions (C1)--(C3). Then $Q$ admits a representation of the form where $r(x) \in [0,1]^d$ is the feature-wise rank-normalized representation, $s : [0,1]^d \rightarrow \mathbb{R}$ is monotone and Lipschitz-continuous, and $\Phi : \mathbb{R} \rightarrow [0,1]$ is monotone and Lipschitz-continuous.

Figures (2)

  • Figure 1: Rank-equivalence invariance under feature-wise strictly increasing transformations. Distinct inputs $x$ and $g(x)$ induce the same rank representation $r(x)$, and thus must be assigned identical outputs by any admissible operator $Q$.
  • Figure 2: Structural constraints induced by conditions (C1)--(C3). Any admissible operator must factor through a rank-based representation followed by a monotone--Lipschitz scalarization.

Theorems & Definitions (17)

  • Theorem 1: Structural Form of Admissible Operators
  • Corollary 1: Minimality of the Axioms
  • Remark 1: Structural Implications
  • Corollary 2: Structural Inadmissibility
  • Proposition 1: Stability of Admissible Operators
  • Lemma 1: Rank-Based Order Preservation
  • proof
  • Lemma 2: Monotone Invariance
  • proof
  • Lemma 3: Batch Independence
  • ...and 7 more