Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Adaptive tail index estimation: minimal assumptions and non-asymptotic guarantees

Johannes Lederer, Anne Sabourin, Mahsa Taheri

TL;DR

This work tackles the challenge of selecting the threshold $k$ for tail index inference using the Hill estimator under minimal assumptions. It introduces Extreme Adaptive Validation (EAV), an adaptive rule on a compact grid that relies on a transparent bias-variance decomposition and explicit variance quantiles, avoiding second-order or von Mises calibrations. Theoretical guarantees show that EAV matches the oracle error up to a small factor and achieves near-minimax rates under von Mises conditions, while remaining robust under only regular variation. Empirically, EAV outperforms existing adaptive schemes on ill-behaved tails and remains competitive on well-behaved tails, with substantially reduced computational complexity due to the grid restriction. Overall, the method provides a practical, assumption-light, and scalable approach for adaptive tail estimation with strong non-asymptotic guarantees.

Abstract

A notoriously difficult challenge in extreme value theory is the choice of the number $k\ll n$, where $n$ is the total sample size, of extreme data points to consider for inference of tail quantities. Existing theoretical guarantees for adaptive methods typically require second-order assumptions or von Mises assumptions that are difficult to verify and often come with tuning parameters that are challenging to calibrate. This paper revisits the problem of adaptive selection of $k$ for the Hill estimator. Our goal is not an `optimal' $k$ but one that is `good enough', in the sense that we strive for non-asymptotic guarantees that might be sub-optimal but are explicit and require minimal conditions. We propose a transparent adaptive rule that does not require preliminary calibration of constants, inspired by `adaptive validation' developed in high-dimensional statistics. A key feature of our approach is the consideration of a grid for $k$ of size $ \ll n $, which aligns with common practice among practitioners but has remained unexplored in theoretical analysis. Our rule only involves an explicit expression of a variance-type term; in particular, it does not require controlling or estimating a biasterm. Our theoretical analysis is valid for all heavy-tailed distributions, specifically for all regularly varying survival functions. Furthermore, when von Mises conditions hold, our method achieves `almost' minimax optimality with a rate of $\sqrt{\log \log n}~ n^{-|ρ|/(1+2|ρ|)}$ when the grid size is of order $\log n$, in contrast to the $ (\log \log (n)/n)^{|ρ|/(1+2|ρ|)} $ rate in existing work. Our simulations show that our approach performs particularly well for ill-behaved distributions.

Adaptive tail index estimation: minimal assumptions and non-asymptotic guarantees

TL;DR

This work tackles the challenge of selecting the threshold for tail index inference using the Hill estimator under minimal assumptions. It introduces Extreme Adaptive Validation (EAV), an adaptive rule on a compact grid that relies on a transparent bias-variance decomposition and explicit variance quantiles, avoiding second-order or von Mises calibrations. Theoretical guarantees show that EAV matches the oracle error up to a small factor and achieves near-minimax rates under von Mises conditions, while remaining robust under only regular variation. Empirically, EAV outperforms existing adaptive schemes on ill-behaved tails and remains competitive on well-behaved tails, with substantially reduced computational complexity due to the grid restriction. Overall, the method provides a practical, assumption-light, and scalable approach for adaptive tail estimation with strong non-asymptotic guarantees.

Abstract

A notoriously difficult challenge in extreme value theory is the choice of the number , where is the total sample size, of extreme data points to consider for inference of tail quantities. Existing theoretical guarantees for adaptive methods typically require second-order assumptions or von Mises assumptions that are difficult to verify and often come with tuning parameters that are challenging to calibrate. This paper revisits the problem of adaptive selection of for the Hill estimator. Our goal is not an `optimal' but one that is `good enough', in the sense that we strive for non-asymptotic guarantees that might be sub-optimal but are explicit and require minimal conditions. We propose a transparent adaptive rule that does not require preliminary calibration of constants, inspired by `adaptive validation' developed in high-dimensional statistics. A key feature of our approach is the consideration of a grid for of size , which aligns with common practice among practitioners but has remained unexplored in theoretical analysis. Our rule only involves an explicit expression of a variance-type term; in particular, it does not require controlling or estimating a biasterm. Our theoretical analysis is valid for all heavy-tailed distributions, specifically for all regularly varying survival functions. Furthermore, when von Mises conditions hold, our method achieves `almost' minimax optimality with a rate of when the grid size is of order , in contrast to the rate in existing work. Our simulations show that our approach performs particularly well for ill-behaved distributions.

Paper Structure

This paper contains 23 sections, 15 theorems, 98 equations, 1 figure, 5 tables.

Table of Contents

  1. Introduction
  2. Adaptive validation
  3. Adaptive Validation framework
  4. Statistical guarantees for EAV
  5. Bias-Variance decomposition for the Hill estimator
  6. Karamata representation of the Hill estimator
  7. Error decomposition
  8. Upper bounds in probability
  9. Additional guarantees under second order conditions
  10. Second order and von Mises conditions, associated exisitng results
  11. A lower bound on $\textcolor{black}{$k^*$}$ and an explicit expression for $n_0(\delta)$
  12. Oracle and adaptive error bounds
  13. Numerical experiments
  14. Discussion
  15. Appendix 1 (Proofs)
  16. ...and 8 more sections

Key Result

Proposition 1

Let $\hat{\gamma}$ satisfy Condition cond:genericErrorDecomp. For any $\delta\in(0,1)$, $n\ge 1$ and $\mathcal{K}$ satisfying Condition cond:wideGrid, we have for all fixed $k ~\le~ \textcolor{black}{$k^*$}(\delta,n)$, with probability at least $1-\delta$,

Figures (1)

  • Figure 1: Monte-Carlo estimates of the standardised RMSE of Hill estimators as a function of the number of order statistics $k$ for samples of size $10\,000$ from the sampling distributions.

Theorems & Definitions (21)

  • Proposition 1: Explicit error bound for the oracle
  • Proposition 2: Optimality properties of $\textcolor{black}{$k^*$}$
  • Remark 1: Interpretation
  • Lemma 1
  • Proposition 3: $\textcolor{black}{\hat{k}_{\operatorname{EAV}}}\ge \textcolor{black}{$k^*$}(\delta_{\mathcal{K}},n)$ with high probability
  • Theorem 1: Error bounds for $\hat{\gamma}(\textcolor{black}{\hat{k}_{\operatorname{EAV}}})$
  • Remark 2: Comparison With The Oracle Error
  • Remark 3: Comparison with boucheron2015tail
  • Lemma 2: An almost sure error decomposition of the Hill estimator
  • Lemma 3: Concentration of $U_{(k+1)}$
  • ...and 11 more