Tree Estimation and Saddlepoint-Based Diagnostics for the Nested Dirichlet Distribution: Application to Compositional Behavioral Data

TL;DR

This work tackles the practical challenges of using the Nested Dirichlet Distribution (NDD) for compositional data by introducing a data-driven greedy tree-finding algorithm and novel diagnostics. It leverages a saddlepoint-based approach to obtain accurate marginal approximations, enabling pseudo-residuals and a likelihood-displacement measure to assess fit and identify influential observations, even when marginals are intractable. The method is demonstrated on synthetic data and Morris water maze results, showing that the learned tree yields interpretable structure and improved fit relative to a standard Dirichlet. An accompanying R package is provided to support reproducible application to new datasets, promising greater adoption of the NDD in practical compositional analyses.

Abstract

The Nested Dirichlet Distribution (NDD) provides a flexible alternative to the Dirichlet distribution for modeling compositional data, relaxing constraints on component variances and correlations through a hierarchical tree structure. While theoretically appealing, the NDD is underused in practice due to two main limitations: the need to predefine the tree structure and the lack of diagnostics for evaluating model fit. This paper addresses both issues. First, we introduce a data-driven, greedy tree-finding algorithm that identifies plausible NDD tree structures from observed data. Second, we propose novel diagnostic tools, including pseudo-residuals based on a saddlepoint approximation to the marginal distributions and a likelihood displacement measure to detect influential observations. These tools provide accurate and computationally tractable assessments of model fit, even when marginal distributions are analytically intractable. We demonstrate our approach through simulation studies and apply it to data from a Morris water maze experiment, where the goal is to detect differences in spatial learning strategies among cognitively impaired and unimpaired mice. Our methods yield interpretable structures and improved model evaluation in a realistic compositional setting. An accompanying R package is provided to support reproducibility and application to new datasets.

Figures (8)

• Figure 1: Comparison of variations of the Nested Dirichlet distribution: (a) standard Dirichlet (b) Nested Dirichlet Distribution (c) Generalized Dirichlet distribution. Each distribution has 5 components.
• Figure 2: Tree diagram example for a NDD with corresponding parameters.
• Figure 3: Tree diagram example for a NDD with corresponding parameters.
• Figure 4: Flow chart of the tree-finding algorithm. The algorithm will fit either a DD or NDD to data using maximum likelihood, AIC or BIC as the criterion.
• Figure 5: Hypothetical example of a top-down, greedy search of tree candidates. In step 1 an initial fit using the standard Dirichlet is considered. Step 2 depicts a better fitting NDD using a single binary split at the root node. Step 3 applies the splitting algorithm on the branch proportions generated from $X_1$,$X_3$, and $X_4$ in which an additional internal node $N_3$ was favored.