Nonparametric Identification of Latent Concepts

TL;DR

This work tackles the problem of identifying latent concepts underlying observed data without parametric generative models. It introduces a nonparametric identifiability framework based on learning by comparison across diverse classes, including both local (pairwise) and global (multi-class) strategies. The authors prove that, under sufficient diversity, class-dependent concepts can be identified up to permutation and invertible transformations, with a nonparametric recovery of the class-concept connective structure, and they extend guarantees to partial identifiability when diversity is incomplete. Empirical results on synthetic and real-world datasets validate the theory, showing improved latent recovery and illustrating the practical reach of the framework.

Abstract

We are born with the ability to learn concepts by comparing diverse observations. This helps us to understand the new world in a compositional manner and facilitates extrapolation, as objects naturally consist of multiple concepts. In this work, we argue that the cognitive mechanism of comparison, fundamental to human learning, is also vital for machines to recover true concepts underlying the data. This offers correctness guarantees for the field of concept learning, which, despite its impressive empirical successes, still lacks general theoretical support. Specifically, we aim to develop a theoretical framework for the identifiability of concepts with multiple classes of observations. We show that with sufficient diversity across classes, hidden concepts can be identified without assuming specific concept types, functional relations, or parametric generative models. Interestingly, even when conditions are not globally satisfied, we can still provide alternative guarantees for as many concepts as possible based on local comparisons, thereby extending the applicability of our theory to more flexible scenarios. Moreover, the hidden structure between classes and concepts can also be identified nonparametrically. We validate our theoretical results in both synthetic and real-world settings.

Figures (21)

• Figure 1: The class "shark " has concepts like "predator ," "sleek body ," and "ocean ." The class "turtle " has concepts like "shell " and "ocean ." A child may learn to distinguish between these classes by focusing on the unique concepts specific to each—such as "predator " and "sleek body " for "shark ," and "shell " for "turtle ."
• Figure 2: The problem setting. Consider images of aquatic animals, where the observed variables $\mathbf{x}$ represent image pixels. The different animal types (e.g., “shark ” and “turtle ”) correspond to class variables $\mathbf{c}$. Class-dependent concept variables $\mathbf{z}_A$ might include attributes like “predator ,” “sleek body ,” “ocean ” and “shell ” (see, e.g., Fig. \ref{['fig:example']}), while class-independent concept variables $\mathbf{z}_B$ could be “lighting ” and “temperature .” The hidden generative process of each image depends on all of these concepts, though only some are specific to each class (encoded by the structure $M$, which is a binary adjacency matrix). The goal is to identify $\mathbf{z}$ based on observed variables $\mathbf{x}$ and classes $\mathbf{c}$.
• Figure 3: Structure in Running Example \ref{['exam:running']}.
• Figure 4: Example of Structural Diversity.
• Figure 5: Identification of class-dependent concepts w.r.t. different numbers of concepts.

Theorems & Definitions (23)

• Example 1
• Example 2
• Definition 1: Observational Equivalence
• Example 3
• Example 4
• Theorem 1: Learning by pairwise comparison
• Corollary 1: Learning by local comparison
• Theorem 2: Learning by global comparison
• Proposition 1: Learning class-independent concepts; Informal
• Proposition 2: Learning class-concept structure