Towards Reliable Simulation-based Inference

TL;DR

This thesis focuses on statistical models that take the form of scientific simulators and provides background about how machine learning can be used for statistical analyses in this context, and shows that Bayesian neural networks can also be used to mitigate the overconfidence of approximations.

Abstract

Scientific knowledge expands by observing the world, hypothesizing some theories about it, and testing them against collected data. When those theories take the form of statistical models, statistical analyses are involved in the process of testing and refining scientific hypotheses. In this thesis, we focus on statistical models that take the form of scientific simulators and provide background about how machine learning can be used for statistical analyses in this context. The first part of this thesis is about showing empirically that performing statistical analyses with machine learning involves a degree of approximation. Specifically, all statistical analyses involve a level of uncertainty in the conclusions drawn, and we show that approximations can lead to overconfident conclusions. We draw caution regarding such overconfident conclusions and introduce a criterion to diagnose overconfident approximations. In the second part, we introduce balancing, a way to regularize machine learning models to reduce overconfidence and favor calibrated or underconfident approximations. Balancing is first introduced for neural ratio estimation algorithms and then extended to other algorithms. Intuition about why balancing leads to less overconfident solutions is provided, and it is shown empirically that balanced algorithms are often either close to calibrated or underconfident. The third part shows that Bayesian neural networks can also be used to mitigate the overconfidence of approximations. Unlike balancing, no regularization is required, and this solution can then work with few training samples and, hence, computationally expensive simulators. To that end, a new Bayesian neural network prior tailored for simulation-based inference is developed, and empirical results show a reduction in overconfidence compared to similar solutions without Bayesian neural networks.

Figures (39)

• Figure 1.1: Visualization of Box's loop box1976scienceblei2014build. The loop iterates over three main elements: model building, model's hidden quantities inference, and model criticism. Iterating over those three elements aims to build better scientific models, taking into account model criticism feedback.
• Figure 2.1: Illustration of the experiment of a feather falling. A gravitational force $\bm\textbf{F} = m\ g\ \bm\textbf{e}$ is applied on the feather and the fall time is measured.
• Figure 2.2: Histogram of samples obtained by running an MCMC algorithm on the feather falling model with measured time $4.68\text{s}$, $6.97\text{s}$ and $9.69\text{s}$ for heights of $3\text{m}$, $5\text{m}$ and $7\text{m}$ respectively.
• Figure 2.3: Approximate posterior density obtained by running a variational inference algorithm on the feather falling model with measured time $4.68\text{s}$, $6.97\text{s}$ and $9.69\text{s}$ for heights of $3\text{m}$, $5\text{m}$ and $7\text{m}$ respectively. The plotted variational distribution is a log normal distribution with parameters $\mu = -1.6873$ and $\sigma = 0.0829$.
• Figure 3.1: Example of feather falling simulations using various gravity and friction parameters. The feather is dropped from the same height in all those configurations, and the feather position is reported at various time steps.

Theorems & Definitions (19)

• Definition 2.1.1
• Definition 2.1.2
• Definition 2.3.1
• Definition 2.3.2
• Definition 2.4.1
• Definition 2.4.2
• Definition 2.4.3
• Definition 3.3.1
• Definition 4.2.1
• Definition 4.2.2