Online and Interactive Bayesian Inference Debugging

TL;DR

The paper tackles the difficulty of debugging Bayesian inference in probabilistic programming by introducing InferLog Holmes, an online and interactive debugger tailored to MCMC workflows. It integrates model visualization, live inference diagnostics, and contextualized warnings within a VSCode extension, enabling real-time analysis during inference. A controlled study with 18 participants shows that the tool increases issue resolution and speeds up identification and iteration, especially on more complex tasks, while reducing overall time spent waiting on inference. The work contributes six design requirements, an open-source implementation, and empirical validation demonstrating that online Bayesian inference debugging can meaningfully accelerate development and improve practitioner success. This approach advances practical debugging in probabilistic programming and lays groundwork for broader PPL integration and future enhancements.

Abstract

Probabilistic programming is a rapidly developing programming paradigm which enables the formulation of Bayesian models as programs and the automation of posterior inference. It facilitates the development of models and conducting Bayesian inference, which makes these techniques available to practitioners from multiple fields. Nevertheless, probabilistic programming is notoriously difficult as identifying and repairing issues with inference requires a lot of time and deep knowledge. Through this work, we introduce a novel approach to debugging Bayesian inference that reduces time and required knowledge significantly. We discuss several requirements a Bayesian inference debugging framework has to fulfill, and propose a new tool that meets these key requirements directly within the development environment. We evaluate our results in a study with 18 experienced participants and show that our approach to online and interactive debugging of Bayesian inference significantly reduces time and difficulty on inference debugging tasks.

Figures (9)

• Figure 1: Example of a suspicious trace plot.
• Figure 2: Warnings Relation. Showing the causes, observable values and triggered warnings.
• Figure 3: Example of warnings raised by the debugger. This example shows one of the debuggers warnings in detail and 2 others collapsed. The warnings try to highlight probable issues and aim to inform the user to a point were they can make a decision if an action is needed or not. Whenever possible warnings show a suggestion that hopefully fixes the underlying issue.
• Figure 4: Exemplified workflows. A) shows the model graph view of the tool right next to the source code. B) is the Live Debugging View, which gives insights into the developing posteriors. C) shows the Details View for variable theta_3 with included pair plots due to the statically identified variance dependence between tau and theta. In D) we see a warning with explanation, graph and the identified line in the source code that causes the issue, together with a suggestion for a solution.
• Figure 5: Forest plot of effect posteriors from Bayesian analysis.