Learning Discrete Concepts in Latent Hierarchical Models
Lingjing Kong, Guangyi Chen, Biwei Huang, Eric P. Xing, Yuejie Chi, Kun Zhang
TL;DR
This work formalizes the problem of learning discrete concepts from high-dimensional data by modeling concepts as discrete latent variables organized in a hierarchical causal graph. It develops identifiability guarantees for recovering bottom-level discrete concepts from continuous observations and for identifying the entire discrete latent hierarchy from the observed distribution, under mild, well-motivated conditions that generalize beyond trees and multi-level DAGs. The authors introduce a rank-based approach, including a discrete analog of non-negative rank and a minimal-graph operator, to recover the latent structure and prove identifiability up to permutation and graphical equivalence. They connect these theoretical insights to latent diffusion models, interpreting diffusion denoising objectives and diffusion steps as recovering concept embeddings at corresponding hierarchical levels, and validate the ideas with synthetic and real diffusion-model experiments. The results offer a principled lens on concept extraction and suggest practical implications for diffusion-based representation learning and causal-sparsity strategies to improve interpretability and controllability.
Abstract
Learning concepts from natural high-dimensional data (e.g., images) holds potential in building human-aligned and interpretable machine learning models. Despite its encouraging prospect, formalization and theoretical insights into this crucial task are still lacking. In this work, we formalize concepts as discrete latent causal variables that are related via a hierarchical causal model that encodes different abstraction levels of concepts embedded in high-dimensional data (e.g., a dog breed and its eye shapes in natural images). We formulate conditions to facilitate the identification of the proposed causal model, which reveals when learning such concepts from unsupervised data is possible. Our conditions permit complex causal hierarchical structures beyond latent trees and multi-level directed acyclic graphs in prior work and can handle high-dimensional, continuous observed variables, which is well-suited for unstructured data modalities such as images. We substantiate our theoretical claims with synthetic data experiments. Further, we discuss our theory's implications for understanding the underlying mechanisms of latent diffusion models and provide corresponding empirical evidence for our theoretical insights.