Flexible Bayesian Multiple Comparison Adjustment Using Dirichlet Process and Beta-Binomial Model Priors

TL;DR

The paper addresses the challenge of adjusting for multiplicity when assessing all possible equality constraints among K groups by placing priors over partitions of the parameter vector. It proposes a flexible beta-binomial prior for partition configurations and compares it to Dirichlet process and uniform priors, using a stochastic search to efficiently navigate the large partition space and enabling model averaging over partitions. The authors contrast multiplicity penalties across priors, conduct extensive simulations to evaluate familywise error rate and power, and demonstrate practical applicability through applications to proportions and means, implemented in the EqualitySampler software. The approach yields coherent, transitive inferences about which groups can be deemed equal and provides shrinkage toward shared values while controlling false positives, offering a Bayesian alternative to classical post-hoc multiple comparison adjustments with explicit probabilistic partition uncertainty.

Abstract

Researchers frequently wish to assess the equality or inequality of groups, but this poses the challenge of adequately adjusting for multiple comparisons. Statistically, all possible configurations of equality and inequality constraints can be uniquely represented as partitions of groups, where any number of groups are equal if they are in the same subset of the partition. In a Bayesian framework, one can adjust for multiple comparisons by constructing a suitable prior distribution over all possible partitions. Inspired by work on variable selection in regression, we propose a class of flexible beta-binomial priors for multiple comparison adjustment. We compare this prior setup to the Dirichlet process prior suggested by Gopalan and Berry (1998) and multiple comparison adjustment methods that do not specify a prior over partitions directly. Our approach not only allows researchers to assess pairwise equality constraints but simultaneously all possible equalities among all groups. Since the space of possible partitions grows rapidly -- for ten groups, there are already 115,975 possible partitions -- we use a stochastic search algorithm to efficiently explore the space. Our method is implemented in the Julia package EqualitySampler, and we illustrate it on examples related to the comparison of means, standard deviations, and proportions.

Figures (12)

• Figure 1: All 52 possible models given $K = 5$, represented as partitions. Circles represent individual parameters and shaded regions indicate which parameters are equal.
• Figure 2: Top: Dirichlet process (left), beta-binomial (middle), and uniform prior (right) across distinct model types for $K = 5$ groups and different prior parameters. Bottom: Same but for the number of inequalities across models.
• Figure 3: The average log prior odds of a partition with 0 inequalities vs. 1 inequality (top left), expected number of clusters (top right), probability of the null model (bottom left), and probability of pairwise equality (bottom right), for various Dirichlet process and beta-binomial priors and the uniform prior for increasing numbers of groups $K$.
• Figure 4: Left: Probability of making at least one false claim about a difference between two groups when there is none. Right: Proportion of falsely claiming no difference between two groups when there is one.
• Figure 5: Familywise error rate across priors and sample sizes under a model with 0 (top left), 1 (top right), 2 (bottom left), and 3 (bottom right) true inequalities for $K = 5$ groups. The rightmost panel shows the average familywise error rate across inequalities.

Theorems & Definitions (10)

• Proposition 1
• proof
• Proposition 2
• proof
• Proposition 3
• proof
• Proposition 4
• proof
• Proposition 5
• proof