Probing the Latent Hierarchical Structure of Data via Diffusion Models

TL;DR

The paper investigates whether natural high-dimensional data harbor a latent hierarchical structure that underpins learnability. It leverages forward-backward diffusion experiments and a Random Hierarchy Model (RHM) to predict a diverging dynamical correlation length and a peak in dynamical susceptibility at a class-reconstruction phase transition. These predictions are validated via Belief Propagation in the RHM and extended to real data, including language (MDLM on WikiText2) and vision (ImageNet with CLIP-tokenized patches), where a finite-time inversion $t^*$ yields system-spanning changes and a susceptibility peak. The work demonstrates that hierarchical latent structure leaves measurable imprints on observable data changes during diffusion, offering a principled tool for probing latent structure, improving interpretability, and informing diffusion-model training strategies. Overall, it argues for the universality of hierarchical, compositional structure in natural data and provides a concrete framework to quantify and exploit it in diffusion-based generative modeling.

Abstract

High-dimensional data must be highly structured to be learnable. Although the compositional and hierarchical nature of data is often put forward to explain learnability, quantitative measurements establishing these properties are scarce. Likewise, accessing the latent variables underlying such a data structure remains a challenge. In this work, we show that forward-backward experiments in diffusion-based models, where data is noised and then denoised to generate new samples, are a promising tool to probe the latent structure of data. We predict in simple hierarchical models that, in this process, changes in data occur by correlated chunks, with a length scale that diverges at a noise level where a phase transition is known to take place. Remarkably, we confirm this prediction in both text and image datasets using state-of-the-art diffusion models. Our results show how latent variable changes manifest in the data and establish how to measure these effects in real data using diffusion models.

Figures (13)

• Figure 1: Example of leaf nodes ${\textnormal{h}}_i^{(0)}$, ${\textnormal{h}}_j^{(0)}$ connected to the common ancestor ${\textnormal{h}}_k^{(\ell)}$ through ${\textnormal{h}}_l^{(\ell-1)}$ and ${\textnormal{h}}_m^{(\ell-1)}$.
• Figure 2: Correlation measures on diffusion samples of the Random Hierarchy Model (RHM).(a-I) In the $\epsilon$-process, the average correlation function shows a correlation length that is maximal for $\epsilon^*\simeq 0.74$, corresponding to the class phase transition, with a system-spanning power-law behavior. The full lines are experiments run with Belief Propagation, while the dashed lines are the corresponding mean-field theory description (Section \ref{['sec:meanfield']}), showing excellent agreement. (a-II) Correspondingly, also the average susceptibility shows a peak at the transition $\epsilon^*$. (b) The same behavior is observed for the correlation function (b-I) and the susceptibility (b-II) for masking diffusion. In this case, the phase transition is observed for inversion time $t^*\simeq 0.3\ T$, where both the correlation length and the susceptibility peak. Data for RHM parameters $v=32$, $m=8$, $s=2$, $L=9$, averaged over $256$ starting data and $256$ diffusion trajectories per starting datum.
• Figure 3: Masking diffusion in the RHM for masking fraction (a)$t/T=0.3$ and (b)$t/T=0.5$. The bottom sequence represents the starting datum ${\bm{x}}_0$. The blue (green) symbols are the masked ones in ${\bm{x}}_t$ that (do not) change feature in $\hat{{\bm{x}}}_0(t)$. The leaves of the tree represent the sampled sequence $\hat{{\bm{x}}}_0(t)\sim p(\hat{{\bm{x}}}_0\vert{\bm{x}}_t)$. In the corresponding tree, the red nodes are those that changed features with respect to the generating tree of ${\bm{x}}_0$. We observe that larger blocks of changed tokens reflect changes in deeper latent variables.
• Figure 4: Forward-backward experiments with language diffusion models.(a) Forward-backward examples for different masking fractions. The words in blue (green) are those that were masked and changed (did not change), while the words in red changed following the backward process. (b) Normalized correlations as a function of index distance $r=|i-j|$ for different fractions of masked tokens. (c) Susceptibility $\chi(t)$ as a function of masking fraction. The results are averaged over $N_S=300$ samples, each consisting of $N_T=128$ tokens, with $N_R=50$ noise realizations for each masking fraction. The susceptibility is given by integrating over the domain $r\in[0,10]$.
• Figure 5: Examples of images generated at different inversion times $t$ with forward-backward diffusion. The first column represents the starting images ${\bm{x}}_0$, while the other columns represent the generated ones $\hat{{\bm{x}}}_0(t)$. The grid indicates the tokens represented inside the CLIP vision encoder. For inversion time $t>0$, the red patches indicate the token embeddings that have a variation magnitude larger than a fixed threshold. These patches of variation appear in domains of growing size around the class transition, observed for $t^*\approx 0.6\div 0.7 T$ (\ref{['fig:clip']}).

Theorems & Definitions (3)

• Definition 1: Token change
• Definition 2: Dynamical correlation function
• Definition 3: Dynamical susceptibility