Learning Human-Aligned Representations with Contrastive Learning and Generative Similarity

TL;DR

The paper addresses learning representations aligned with human cognition by integrating a Bayesian notion of generative similarity into contrastive learning. Generative similarity $s_{\text{gen}}$ is defined as the Bayes odds ratio for two samples coming from the same distribution, and is incorporated into a contrastive objective to shape embeddings toward human-like structure. Across shape regularity, abstract Euclidean geometry via probabilistic programs (Geoclidean), and hierarchical ImageNet categories, GenSim improves alignment with human behavior over standard contrastive or supervised baselines. The results demonstrate closed-form and programmatic generative similarities that enable few-shot generalization and hierarchical decoding, offering a scalable path to human-centric representations with domain-informed priors. This approach reduces reliance on large-scale human judgments while preserving the ability to capture nuanced cognitive structure.

Abstract

Humans rely on effective representations to learn from few examples and abstract useful information from sensory data. Inducing such representations in machine learning models has been shown to improve their performance on various benchmarks such as few-shot learning and robustness. However, finding effective training procedures to achieve that goal can be challenging as psychologically rich training data such as human similarity judgments are expensive to scale, and Bayesian models of human inductive biases are often intractable for complex, realistic domains. Here, we address this challenge by leveraging a Bayesian notion of generative similarity whereby two data points are considered similar if they are likely to have been sampled from the same distribution. This measure can be applied to complex generative processes, including probabilistic programs. We incorporate generative similarity into a contrastive learning objective to enable learning of embeddings that express human cognitive representations. We demonstrate the utility of our approach by showing that it can be used to capture human-like representations of shape regularity, abstract Euclidean geometric concepts, and semantic hierarchies for natural images.

Figures (7)

• Figure 1: Schematic representation of generative similarity.A. Graphical models for the same and different data generation hypotheses. B. Example same and different quadrilateral shape pairs.
• Figure 2: Generative similarity can instill human shape regularity biases.A. The oddball task of sable2021sensitivity used six quadrilateral stimulus images, in which five images were of the same reference shape (differing in scale and rotation) and one was an oddball (highlighted in red) that diverged from the reference shape’s geometric properties. In this example, the reference shape is a rectangle; note that the oddball does not have four right angles like the rectangles. B. Mean error rates per quadrilateral type on the oddball task, ordered from most regular to least regular shapes. sable2021sensitivity found that human error rates systematically increase as the regularity of the reference quadrilaterals decrease, whereas those of non-human primates do not. We use this same evaluation for a CNN model with different finetuning objectives, reporting the Spearman rank correlation between model performance and number of geometric regularities across quadrilateral type (see Table 1 in Appendix \ref{['app:quad-detail']} for these values). Only the model with the generative similarity (GenSim) finetuning objective exhibited a significant human geometric regularity effect. Error bars denote standard errors over $10$ training runs, red line shows chance-level error rate. C. Correlation between model error rates and human or monkey error rates. Supervised is the most monkey-like whereas GenSim is the most human-like. D. We show the Spearman rank correlation of pairwise human similarity judgements of quadrilaterals to those of the models.
• Figure 3: Generative similarity over probabilistic programs helps contrastive learning models better represent geometric concepts.A. Example program, program parse tree, and rendered generations from the Geoclidean DSL. B. In the Geoclidean classification task, participants or models are given examples from a reference program. They have to successfully distinguish between examples from the reference program and two types of negative examples - ones from a "close" program and ones from a "far" program. C. Performance of contrastive embeddings in correctly classifying reference program images. Green lines indicate mean human accuracy and errorbars denote standard error over training runs.
• Figure 4: Generative Similarity Helps Encode Natural Image HierarchiesA. WordNet church1990word gives a hierarchy that spans multiple semantic levels. The object categories in the classic ImageNet dataset deng2009imagenet are sampled from different places on the WordNet tree. B. Mean classification accuracy over chance across tree levels. Error bars denote standard errors over model training runs. See Supplementary Figure \ref{['fig:tree_level_acc']} for accuracies on individual tree levels. C. The Levels Dataset of muttenthaler2024improving collected human odd-one-out judgments at $3$ different abstraction levels where humans are given three images and choose what they believe is the odd-one-out. D. Mean model likelihood across human participants calculated using each model's pairwise embeddings' similarities. We found that the GenSim model best accounts for human behavior. Error bars are standard errors across human subjects.
• Figure S1: Encoding the generative similarity of a Gaussian mixture.A. Optimal linear projection vector $\varphi$ for a symmetric Gaussian mixture with means $\pm\mu$. B. Learned 1D embedding values from a two-layer perceptron for points sampled from a 2D Gaussian mixture (colors indicate values). C. Mean generative similarity as a function of distance in the embedding space shown in B (discretized into 500 quantile bins). Shaded area indicates 95% CIs bootsrapped over data points.