Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Confidence and Assurance of Percentiles

Sanjay M. Joshi

TL;DR

The paper addresses the lack of rigorous confidence qualifiers for percentile and tolerance statements by deriving a distribution-free, binomial-based expression for the confidence in the $p$-th percentile using order statistics, and by introducing tolerance and assurance intervals that pair percentile information with a desired confidence. It provides closed-form-like expressions via the binomial CDF $c = \mathrm{CDF}(j-1;n,p)$ and extends the framework to central tolerance intervals, as well as to assurance intervals computed with Brent's method. An open-source Python package, joshi23, accompanies the work, enabling practical calculation and demonstration through Jupyter notebooks. Overall, the work offers distribution-agnostic, interpretable tools to communicate percentile uncertainty and tolerance in applied settings, and highlights how median and mean confidence intervals can behave differently depending on the underlying distribution.

Abstract

Confidence interval of mean is often used when quoting statistics. The same rigor is often missing when quoting percentiles and tolerance or percentile intervals. This article derives the expression for confidence in percentiles of a sample population. Confidence intervals of median is compared to those of mean for a few sample distributions. The concept of assurance from reliability engineering is then extended to percentiles. The assurance level of sorted samples simply matches the confidence and percentile levels. Numerical method to compute assurance using Brent's optimization method is provided as an open-source python package.

Confidence and Assurance of Percentiles

TL;DR

The paper addresses the lack of rigorous confidence qualifiers for percentile and tolerance statements by deriving a distribution-free, binomial-based expression for the confidence in the -th percentile using order statistics, and by introducing tolerance and assurance intervals that pair percentile information with a desired confidence. It provides closed-form-like expressions via the binomial CDF and extends the framework to central tolerance intervals, as well as to assurance intervals computed with Brent's method. An open-source Python package, joshi23, accompanies the work, enabling practical calculation and demonstration through Jupyter notebooks. Overall, the work offers distribution-agnostic, interpretable tools to communicate percentile uncertainty and tolerance in applied settings, and highlights how median and mean confidence intervals can behave differently depending on the underlying distribution.

Abstract

Confidence interval of mean is often used when quoting statistics. The same rigor is often missing when quoting percentiles and tolerance or percentile intervals. This article derives the expression for confidence in percentiles of a sample population. Confidence intervals of median is compared to those of mean for a few sample distributions. The concept of assurance from reliability engineering is then extended to percentiles. The assurance level of sorted samples simply matches the confidence and percentile levels. Numerical method to compute assurance using Brent's optimization method is provided as an open-source python package.
Paper Structure (5 sections, 9 equations, 4 figures)

This paper contains 5 sections, 9 equations, 4 figures.

Table of Contents

  1. Introduction
  2. Confidence interval of percentiles
  3. Confidence in tolerance interval
  4. Assurance interval for sorted samples
  5. Conclusions

Figures (4)

  • Figure 1: Spread of confidence interval of median vs. number of samples at different levels of confidence levels
  • Figure 2: Median and mean with 95% confidence intervals shown as shaded area for a normal variable ($\mu = 0$, $\sigma = 1$) at different number of samples
  • Figure 3: KDE plots of 95% confidence intervals of median against 95% confidence intervals of mean for different distributions
  • Figure 4: Tolerance Intervals with confidence and Assurance Intervals for 200 sorted samples