Mixture-of-experts Wishart model for covariance matrices with an application to Cancer drug screening

TL;DR

A comprehensive Bayesian framework for analyzing heterogeneous covariance data through both classical mixture models and a novel mixture-of-experts Wishart model, illustrating the ability of the MoE-Wishart model to leverage covariance across drug dosages and replicate measurements.

Abstract

Covariance matrices arise naturally in different scientific fields, including finance, genomics, and neuroscience, where they encode dependence structures and reveal essential features of complex multivariate systems. In this work, we introduce a comprehensive Bayesian framework for analyzing heterogeneous covariance data through both classical mixture models and a novel mixture-of-experts Wishart (MoE-Wishart) model. The proposed MoE-Wishart model extends standard Wishart mixtures by allowing mixture weights to depend on predictors through a multinomial logistic gating network. This formulation enables the model to capture complex, nonlinear heterogeneity in covariance structures and to adapt subpopulation membership probabilities to covariate-dependent patterns. To perform inference, we develop an efficient Gibbs-within-Metropolis-Hastings sampling algorithm tailored to the geometry of the Wishart likelihood and the gating network. We additionally derive an Expectation-Maximization algorithm for maximum likelihood estimation in the mixture-of-experts setting. Extensive simulation studies demonstrate that the proposed Bayesian and maximum likelihood estimators achieve accurate subpopulation recovery and estimation under a range of heterogeneous covariance scenarios. Finally, we present an innovative application of our methodology to a challenging dataset: cancer drug sensitivity profiles, illustrating the ability of the MoE-Wishart model to leverage covariance across drug dosages and replicate measurements. Our methods are implemented in the \texttt{R} package \texttt{moewishart} available at https://github.com/zhizuio/moewishart .

Figures (13)

• Figure 1: Simulation results. Data were generated from finite mixture models over 100 Monte Carlo replications. Results are reported for sample sizes $n \in \{200, 500, 1000\}$ under two dimensional settings: (A) $p=2$ and (B) $p=8$. The four competing methods compared include: (i) Bayesian mixture models (Bayes); (ii) EM-based mixture models (EM); (iii) Bayesian mixture-of-experts models (Bayes-MoE); and (iv) EM-based mixture-of-experts models (EM-MoE). Estimation performance is evaluated using average componentwise errors: $\frac{1}{K}\|\hat{\bm\pi}-\bm\pi\|_1 ; \; \frac{1}{K}\|\hat{\bm\nu}-\bm\nu\|_1 ; \; \frac{1}{K}\sum_{k=1}^K\|\hat{\Sigma}_k-\Sigma_k\|_2^2 .$
• Figure 2: Simulation results. Data were generated from the mixture-of-experts (MoE) model across 100 simulated data sets. Panels (A) and (B) correspond to the settings $p=2$ and $p=8$, respectively. The four competing methods compared are: (i) Bayesian inference under the mixture model (Bayes); (ii) EM-based inference under the mixture model (EM); (iii) Bayesian inference under the mixture-of-experts model (Bayes-MoE); and (iv) EM-based inference under the mixture-of-experts model (EM-MoE).
• Figure 3: Experimental setup and data for cell lines from patient biopsies treated with drugs at multiple concentrations. (A) Experiment setup for an individual cell line experiment in multi-well cell culture plates. (B) Drug dose/concentration–response relationship for data from one row of wells of the plate. AUC is the area under the dose–response curve, often obtained from a fitted sigmoid function. IC$_{50}$ (half-maximal inhibitory concentration) represents the concentration of a drug required to reduce cell viability (survival) by $50\%$ compared to a control.
• Figure 4: Drug clustering based on sensitivity data. The heatmap shows AUC values for 370 cell lines treated with 172 drugs. AUCs were standardized and subjected to hierarchical clustering (complete linkage method with Euclidean distance) to assess similarity among compounds. The legend "Model cluster index" shows the cluster labels from the four methods. As shown, AUC-derived clusters are not aligned with known mechanism-of-actions (MoA), whereas the Bayesian MoE model yields more coherent mechanism-based groupings.
• Figure 5: Alluvial diagrams showing flows from MoA groups into individual model-based clusters, illustrating cluster-MoA cohesion. Purity in each panel is defined as: $\text{purity} = \frac{1}{n}\sum_{k=1}^{K} \max_{c} | \{ i : \text{cluster}(i)=k \land \text{MoA\_group}(i)=c \} |$.