Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Graphical model-based clustering of categorical data

Laura Ferrini, Federico Castelletti

TL;DR

The paper develops a Dirichlet Process mixture of categorical graphical models to cluster multivariate categorical data while explicitly modeling cluster-specific dependence structures via decomposable graphs. By placing a Hyper-Dirichlet prior on θ given G and a graph prior on G, and using a DP prior over atoms (θ,G), the method learns both cluster assignments and cluster-specific graphical structures in a fully Bayesian manner. A partially collapsed MCMC yields posterior similarity matrices and edge-inclusion probabilities, enabling interpretable, dependence-aware clustering demonstrated on simulated data and real-world datasets (genomics and voting records). The approach improves over independence-assuming methods when clusters differ in dependence patterns, though it faces computational challenges due to graph-space exploration and motivates scalable extensions such as variational or hierarchical nonparametric schemes.

Abstract

Clustering multivariate data is a pervasive task in many applied problems, particularly in social studies and life science. Model-based approaches to clustering rely on mixture models, where each mixture component corresponds to the kernel of a distribution characterizing a latent sub-group. Current methods developed within this framework employ multivariate distributions built under the assumption of independence among variables given the cluster allocation. Accordingly, possible dependence structures characterizing differences across groups are not directly accounted for during the clustering process. In this paper we consider multivariate categorical data, and introduce a model-based clustering method which employs graphical models as a tool to encode dependencies between variables. Specifically, we consider a Dirichlet Process mixture of categorical graphical models, which clusters individuals into groups that are homogeneous in terms of dependence (graphical) structure and allied parameters. We provide full Bayesian inference for the model and develop a Markov chain Monte Carlo scheme for posterior analysis. Our method is evaluated through simulations and applied to real case studies, including the analysis of genomic data and voting records. Results reveal the merits of a graphical model-based clustering, in comparison with approaches that do not explicitly account for dependencies in the multivariate distribution of variables.

Graphical model-based clustering of categorical data

TL;DR

The paper develops a Dirichlet Process mixture of categorical graphical models to cluster multivariate categorical data while explicitly modeling cluster-specific dependence structures via decomposable graphs. By placing a Hyper-Dirichlet prior on θ given G and a graph prior on G, and using a DP prior over atoms (θ,G), the method learns both cluster assignments and cluster-specific graphical structures in a fully Bayesian manner. A partially collapsed MCMC yields posterior similarity matrices and edge-inclusion probabilities, enabling interpretable, dependence-aware clustering demonstrated on simulated data and real-world datasets (genomics and voting records). The approach improves over independence-assuming methods when clusters differ in dependence patterns, though it faces computational challenges due to graph-space exploration and motivates scalable extensions such as variational or hierarchical nonparametric schemes.

Abstract

Clustering multivariate data is a pervasive task in many applied problems, particularly in social studies and life science. Model-based approaches to clustering rely on mixture models, where each mixture component corresponds to the kernel of a distribution characterizing a latent sub-group. Current methods developed within this framework employ multivariate distributions built under the assumption of independence among variables given the cluster allocation. Accordingly, possible dependence structures characterizing differences across groups are not directly accounted for during the clustering process. In this paper we consider multivariate categorical data, and introduce a model-based clustering method which employs graphical models as a tool to encode dependencies between variables. Specifically, we consider a Dirichlet Process mixture of categorical graphical models, which clusters individuals into groups that are homogeneous in terms of dependence (graphical) structure and allied parameters. We provide full Bayesian inference for the model and develop a Markov chain Monte Carlo scheme for posterior analysis. Our method is evaluated through simulations and applied to real case studies, including the analysis of genomic data and voting records. Results reveal the merits of a graphical model-based clustering, in comparison with approaches that do not explicitly account for dependencies in the multivariate distribution of variables.
Paper Structure (14 sections, 2 theorems, 22 equations, 7 figures)

This paper contains 14 sections, 2 theorems, 22 equations, 7 figures.

Table of Contents

  1. Introduction
  2. Background
  3. Categorical data distributions
  4. Graphical models
  5. Mixtures of categorical graphical models
  6. Baseline over graph-parameters
  7. Baseline over graphs
  8. Posterior inference
  9. Posterior summaries
  10. Simulation study
  11. Real data analyses
  12. Cystic fibrosis data
  13. Voting records
  14. Conclusion

Key Result

Proposition 4.1

The posterior predictive distribution of $\bm{x}^{(i)}$, given all the other observations currently assigned to cluster $k$, $\{\boldsymbol{x}^{(l)} : l \neq i, \xi_l = k\}$, and the graph $\mathcal{G}_k$ having set of cliques and separators $\mathcal{C}_k$ and $\mathcal{S}_k$ respectively is such t where, for a generic $S \subseteq V$, $a_{\bm{x}^{(i)}_S}^S$ denotes the element of $\boldsymbol{a}

Figures (7)

  • Figure 1: A decomposable graph $\mathcal{G}$ on the set of vertices $V=\{1,\dots,6\}$. $\mathcal{G}$ has the set of cliques $\mathcal{C}=\{C_1,C_2,C_3,C_4\}$ with $C_1=\{1,2\},C_2=\{2,3,5\},C_3=\{2,4,5\},C_4=\{5,6\}$, and set of separators $\mathcal{S}=\{S_2,S_3,S_4\}$, with $S_2=\{2\},S_3=\{2,5\},S_3=\{5\}$.
  • Figure 2: Simulations. Distribution of the Variation of Information (VI) between true and estimated clusterings for values of $q\in\{10,20\}$ and each method under comparison. The three scenarios reflect different degrees of dependence and similarity between the two cluster-specific graphs: Scenario 0 is characterized by equal graphs with no edges; Scenario 1 and Scenario 2 by graphical structures having distance $M=10$ and $M=20$ respectively.
  • Figure 3: Cystic fibrosis data. Estimated posterior similarity matrix, with subjects arranged according to the estimated clustering partition, and colors representing their disease status, red and green for cases and controls respectively.
  • Figure 4: Cystic fibrosis data. Estimated posterior probabilities of edge inclusion for two randomly chosen individuals within each of the three estimated clusters (one row for each cluster).
  • Figure 5: Cystic fibrosis data. Graph representation of estimated subject-specific posterior probabilities of edge inclusion representing the dependence structure between loci. Each graph refers to a subject randomly selected from one of the three estimated clusters.
  • ...and 2 more figures

Theorems & Definitions (4)

  • Proposition 4.1: Posterior predictive distribution - non-empty cluster
  • proof
  • Proposition 4.2: Prior predictive distribution - empty cluster
  • proof