Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A Statistical Decision-Theoretical Perspective on the Two-Stage Approach to Parameter Estimation

Braghadeesh Lakshminarayanan, Cristian R. Rojas

TL;DR

This paper addresses parameter estimation when the likelihood is difficult to evaluate by situating the Two-Stage (TS) approach within statistical decision theory. It derives Bayes and minimax estimators for TS, and specializes the method to i.i.d. data with a quantile-based first stage and a convex, linear-second stage to enable efficient optimization. The authors implement Monte Carlo approximations and importance sampling to derive practical rules, and demonstrate the methods on Weibull parameter estimation, showing strong performance for the scale parameter and more sensitivity for the shape parameter to feature design and priors. The work provides a formal justification for TS and a tractable optimization framework, suggesting that careful feature engineering and prior choices can yield robust and efficient estimators in settings where simulation is feasible but the likelihood is not.

Abstract

One of the most important problems in system identification and statistics is how to estimate the unknown parameters of a given model. Optimization methods and specialized procedures, such as Empirical Minimization (EM) can be used in case the likelihood function can be computed. For situations where one can only simulate from a parametric model, but the likelihood is difficult or impossible to evaluate, a technique known as the Two-Stage (TS) Approach can be applied to obtain reliable parametric estimates. Unfortunately, there is currently a lack of theoretical justification for TS. In this paper, we propose a statistical decision-theoretical derivation of TS, which leads to Bayesian and Minimax estimators. We also show how to apply the TS approach on models for independent and identically distributed samples, by computing quantiles of the data as a first step, and using a linear function as the second stage. The proposed method is illustrated via numerical simulations.

A Statistical Decision-Theoretical Perspective on the Two-Stage Approach to Parameter Estimation

TL;DR

This paper addresses parameter estimation when the likelihood is difficult to evaluate by situating the Two-Stage (TS) approach within statistical decision theory. It derives Bayes and minimax estimators for TS, and specializes the method to i.i.d. data with a quantile-based first stage and a convex, linear-second stage to enable efficient optimization. The authors implement Monte Carlo approximations and importance sampling to derive practical rules, and demonstrate the methods on Weibull parameter estimation, showing strong performance for the scale parameter and more sensitivity for the shape parameter to feature design and priors. The work provides a formal justification for TS and a tractable optimization framework, suggesting that careful feature engineering and prior choices can yield robust and efficient estimators in settings where simulation is feasible but the likelihood is not.

Abstract

One of the most important problems in system identification and statistics is how to estimate the unknown parameters of a given model. Optimization methods and specialized procedures, such as Empirical Minimization (EM) can be used in case the likelihood function can be computed. For situations where one can only simulate from a parametric model, but the likelihood is difficult or impossible to evaluate, a technique known as the Two-Stage (TS) Approach can be applied to obtain reliable parametric estimates. Unfortunately, there is currently a lack of theoretical justification for TS. In this paper, we propose a statistical decision-theoretical derivation of TS, which leads to Bayesian and Minimax estimators. We also show how to apply the TS approach on models for independent and identically distributed samples, by computing quantiles of the data as a first step, and using a linear function as the second stage. The proposed method is illustrated via numerical simulations.
Paper Structure (12 sections, 25 equations, 6 figures, 1 table)

This paper contains 12 sections, 25 equations, 6 figures, 1 table.

Table of Contents

  1. Introduction
  2. Problem Formulation
  3. A Statistical Decision-Theoretical Formulation of TS
  4. Bayes estimator
  5. Minimax estimator
  6. Two-Stage Approach for i.i.d. Data
  7. First Stage
  8. Second stage
  9. Simulations
  10. Bayes estimator
  11. Minimax estimator
  12. Conclusions

Figures (6)

  • Figure 1: Scatter plot (blue dots) of Bayes TS estimates of $\eta$ vs. its true value for a uniform prior. The red dashed line corresponds to an oracle estimate, which knows the true value of the parameter.
  • Figure 2: Scatter plot (blue dots) of Bayes TS estimates of $\gamma$ vs. its true value for a uniform prior. The red dashed line corresponds to an oracle estimate, which knows the true value of the parameter.
  • Figure 3: Scatter plot (blue dots) of Bayes TS estimates of $\eta$ vs. its true value for an uninformative prior. The red dashed line corresponds to an oracle estimate, which knows the true value of the parameter.
  • Figure 4: Scatter plot (blue dots) of Bayes TS estimates of $\gamma$ vs. its true value for an uninformative prior. The red dashed line corresponds to an oracle estimate, which knows the true value of the parameter.
  • Figure 5: Scatter plot (blue dots) of minimax TS estimates of $\eta$ vs. its true value. The red dashed line corresponds to an oracle estimate, which knows the true value of the parameter.
  • ...and 1 more figures

Theorems & Definitions (1)

  • Remark III.1