Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Statistical significance in choice modelling: computation, usage and reporting

Stephane Hess, Andrew Daly, Michiel Bliemer, Angelo Guevara, Ricardo Daziano, Thijs Dekker

TL;DR

There is an over-reliance on 95\% confidence levels, misunderstandings of the meaning of significance, and a lack of precision in the reporting of measures of uncertainty in many studies, especially when using $p-values and even more so with \emph{star} measures.

Abstract

This paper offers a commentary on the use of notions of statistical significance in choice modelling. We review the reasons for uncertainty in parameter estimates, provide a precise discussion on the computation of measures of uncertainty and confidence intervals, and discuss the use of statistical tests. We argue that, as in many other areas of science, there is an over-reliance on 95\% confidence levels, and misunderstandings of the meaning of significance. We also observe a lack of precision in the reporting of measures of uncertainty in many studies, especially when using $p$-values and even more so with \emph{star} measures. The paper also stresses the importance of considering behavioural or policy significance in addition to statistical significance. Finally, we stress a number of points that are specific to choice modelling and which require special attention, notably in relation to derived measures such as willingness-to-pay, the treatment of random heterogeneity, and the use of repeated choice data.

Statistical significance in choice modelling: computation, usage and reporting

TL;DR

There is an over-reliance on 95\% confidence levels, misunderstandings of the meaning of significance, and a lack of precision in the reporting of measures of uncertainty in many studies, especially when using $p-values and even more so with \emph{star} measures.

Abstract

This paper offers a commentary on the use of notions of statistical significance in choice modelling. We review the reasons for uncertainty in parameter estimates, provide a precise discussion on the computation of measures of uncertainty and confidence intervals, and discuss the use of statistical tests. We argue that, as in many other areas of science, there is an over-reliance on 95\% confidence levels, and misunderstandings of the meaning of significance. We also observe a lack of precision in the reporting of measures of uncertainty in many studies, especially when using -values and even more so with \emph{star} measures. The paper also stresses the importance of considering behavioural or policy significance in addition to statistical significance. Finally, we stress a number of points that are specific to choice modelling and which require special attention, notably in relation to derived measures such as willingness-to-pay, the treatment of random heterogeneity, and the use of repeated choice data.

Paper Structure

This paper contains 21 sections, 10 equations, 2 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Measures of uncertainty
  3. Sampling error and parameter uncertainty
  4. From maximum likelihood estimation to the asymptotic covariance matrix
  5. Robust covariance matrix and bootstrapping
  6. Error measures for parameter transformations
  7. Repeated choice data
  8. Identification and over-specification issues
  9. Confidence intervals
  10. Hypothesis tests
  11. Formulation of hypotheses and possible test outcomes
  12. Hypotheses for tests on individual parameters
  13. Hypotheses for tests comparing models
  14. A trinity of tests
  15. Considerations specific to choice models
  16. ...and 6 more sections

Figures (2)

  • Figure 1: Graphical representation of the calculation of asymptotic confidence intervals
  • Figure 2: Relationship between estimates and log-likelihood values used for likelihood ratio, Wald, and Lagrange multiplier tests