Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Statistical Verification of Linear Classifiers

Anton Zhiyanov, Alexander Shklyaev, Alexey Galatenko, Vladimir Galatenko, Alexander Tonevitsky

TL;DR

This paper introduces a statistical framework to verify that linear classifiers capture real differences between two samples rather than performing by chance. It develops upper bounds on the p-value for near-linear separability in two dimensions and extends these results to integral, conditional, and permutation-based testing, with special emphasis on normally distributed data. The authors build two homogeneity tests—the Linear test and the Angle test—and provide practical algorithms, including exact, approximate, and SVM-based p-value computations, validated on synthetic data. They apply the method to ER-positive breast cancer recurrence classifiers, showing the IGFBP6–ELOVL5 gene pair exhibits meaningful non-random separation, thereby supporting its biological relevance despite potential multiple-testing concerns. The work offers a rigorous, scalable approach for validating disease biomarker classifiers in the presence of data heterogeneity and small testing sets, with open-source tools for broader adoption.

Abstract

We propose a homogeneity test closely related to the concept of linear separability between two samples. Using the test one can answer the question whether a linear classifier is merely ``random'' or effectively captures differences between two classes. We focus on establishing upper bounds for the test's \emph{p}-value when applied to two-dimensional samples. Specifically, for normally distributed samples we experimentally demonstrate that the upper bound is highly accurate. Using this bound, we evaluate classifiers designed to detect ER-positive breast cancer recurrence based on gene pair expression. Our findings confirm significance of IGFBP6 and ELOVL5 genes in this process.

Statistical Verification of Linear Classifiers

TL;DR

This paper introduces a statistical framework to verify that linear classifiers capture real differences between two samples rather than performing by chance. It develops upper bounds on the p-value for near-linear separability in two dimensions and extends these results to integral, conditional, and permutation-based testing, with special emphasis on normally distributed data. The authors build two homogeneity tests—the Linear test and the Angle test—and provide practical algorithms, including exact, approximate, and SVM-based p-value computations, validated on synthetic data. They apply the method to ER-positive breast cancer recurrence classifiers, showing the IGFBP6–ELOVL5 gene pair exhibits meaningful non-random separation, thereby supporting its biological relevance despite potential multiple-testing concerns. The work offers a rigorous, scalable approach for validating disease biomarker classifiers in the presence of data heterogeneity and small testing sets, with open-source tools for broader adoption.

Abstract

We propose a homogeneity test closely related to the concept of linear separability between two samples. Using the test one can answer the question whether a linear classifier is merely ``random'' or effectively captures differences between two classes. We focus on establishing upper bounds for the test's \emph{p}-value when applied to two-dimensional samples. Specifically, for normally distributed samples we experimentally demonstrate that the upper bound is highly accurate. Using this bound, we evaluate classifiers designed to detect ER-positive breast cancer recurrence based on gene pair expression. Our findings confirm significance of IGFBP6 and ELOVL5 genes in this process.
Paper Structure (16 sections, 9 theorems, 37 equations, 3 figures, 1 table)

This paper contains 16 sections, 9 theorems, 37 equations, 3 figures, 1 table.

Table of Contents

  1. Introduction
  2. Results
  3. Conditional bounds
  4. Integral bounds
  5. Homogeneity test
  6. Permutational test
  7. Synthetic experiments
  8. Application to breast cancer recurrence data
  9. Materials and methods
  10. ER-positive breast cancer recurrence
  11. Implementation details
  12. Discussion
  13. Proofs
  14. Funding
  15. Conflict of interest
  16. ...and 1 more sections

Key Result

Lemma 1

Denote by $\mathcal{A}_{\leq m} = \bigcup_{h \leq m} \mathcal{A}_h$. Then, for $k \geq 2 m - 1$,

Figures (3)

  • Figure 1: A: The upper bound \ref{['eq:main-integral-normal']} of $\IfNoValueTF{-NoValue-} { \IfNoValueTF{ \mathcal{A}_0 } { \mathbf{P} } { \mathbf{P} \left( \mathcal{A}_0 \right) } } { \mathbf{P} \left( \left. \mathcal{A}_0 \right| -NoValue- \right) }$ in the normal case is tight. B: The upper bound from Lemma \ref{['lem:error-bound']} approximates $\IfNoValueTF{-NoValue-} { \IfNoValueTF{ \mathcal{A}_{\leq m} } { \mathbf{P} } { \mathbf{P} \left( \mathcal{A}_{\leq m} \right) } } { \mathbf{P} \left( \left. \mathcal{A}_{\leq m} \right| -NoValue- \right) }$ reasonably well.
  • Figure 2: The power of Permutational test A: for the normal samples $\mathcal{N}(0, E)$ and $\mathcal{N}((0, \mu), E)$, B: for the uniform samples $\mathcal{U}[0, 1]^2$ and $\mathcal{U}[\mu, \mu + 1] \times [0, 1]$, where $\mu \geq 0$.
  • Figure 3: A: The recurrent patients of ER-positive breast cancer differ significantly from non-recurrent ones, i.e., the constructed classifier is not "random", p-value $\approx 0.002 < 0.05$. B: The classifier constructed on permuted labels is "random", since p-value $\approx 0.76 > 0.05$ (i.e., the number of errors is too high to reject the homogeneity hypothesis $\mathrm{H}_0$).

Theorems & Definitions (25)

  • Definition 1
  • Lemma 1
  • Lemma 2
  • Theorem 1
  • Definition 2
  • Theorem 2
  • Theorem 3
  • Remark 1
  • Corollary 1: Linear test
  • Remark 2
  • ...and 15 more