Bayesian shrinkage priors subject to linear constraints
Zhi Ling, Shozen Dan
TL;DR
The paper tackles identifiability and interpretability in Bayesian regression with full K-level categorical predictors by enforcing a sum-to-zero constraint. It introduces a unified multivariate normal prior that respects arbitrary linear constraints with a closed-form mean $m^*$ and covariance $\Sigma^*$, and provides an efficient projection-based sampling algorithm that works across shrinkage priors such as Bayesian ridge and horseshoe. The authors derive explicit constrained covariances for common priors, propose a low-dimensional null-space sampling approach, and supply ready-to-use sum-to-zero specializations for Bayesian ridge, hierarchical Bayesian ridge, and horseshoe variants. This framework enables principled uncertainty quantification, preserves marginal priors as much as possible, and facilitates implementation in probabilistic programming environments for constrained Bayesian inference.
Abstract
In Bayesian regression models with categorical predictors, constraints are needed to ensure identifiability when using all $K$ levels of a factor. The sum-to-zero constraint is particularly useful as it allows coefficients to represent deviations from the population average. However, implementing such constraints in Bayesian settings is challenging, especially when assigning appropriate priors that respect these constraints and general principles. Here we develop a multivariate normal prior family that satisfies arbitrary linear constraints while preserving the local adaptivity properties of shrinkage priors, with an efficient implementation algorithm for probabilistic programming languages. Our approach applies broadly to various shrinkage frameworks including Bayesian Ridge, horseshoe priors and their variants, demonstrating excellent performance in simulation studies. The covariance structure we derive generalizes beyond regression models to any Bayesian analysis requiring linear constraints on parameters, providing practitioners with a principled approach to parameter identification while maintaining proper uncertainty quantification and interpretability.