Tversky Neural Networks: Psychologically Plausible Deep Learning with Differentiable Tversky Similarity

TL;DR

This work argues that standard geometric similarity is psychologically implausible and introduces a differentiable, Tversky-based similarity to ground deep learning in human-inspired feature matching. It formalizes a dual representation of objects, defines differentiable measures for common and distinctive features, and builds two neural modules: the Tversky Similarity Layer and the Tversky Projection Layer, with data-domain visualization to aid interpretability. Across XOR, PTB language modeling with GPT-2 variants, and vision tasks with ResNet-50, the approach yields competitive or improved accuracy and perplexity, often with fewer parameters when prototypes are shared through feature banks. The framework offers a principled path to psychologically plausible, interpretable deep learning, with potential for targeted model editing via semantic fields and data-space prototypes.

Abstract

Work in psychology has highlighted that the geometric model of similarity standard in deep learning is not psychologically plausible because its metric properties such as symmetry do not align with human perception of similarity. In contrast, Tversky (1977) proposed an axiomatic theory of similarity with psychological plausibility based on a representation of objects as sets of features, and their similarity as a function of their common and distinctive features. This model of similarity has not been used in deep learning before, in part because of the challenge of incorporating discrete set operations. In this paper, we develop a differentiable parameterization of Tversky's similarity that is learnable through gradient descent, and derive basic neural network building blocks such as the Tversky projection layer, which unlike the linear projection layer can model non-linear functions such as XOR. Through experiments with image recognition and language modeling neural networks, we show that the Tversky projection layer is a beneficial replacement for the linear projection layer. For instance, on the NABirds image classification task, a frozen ResNet-50 adapted with a Tversky projection layer achieves a 24.7% relative accuracy improvement over the linear layer adapter baseline. With Tversky projection layers, GPT-2's perplexity on PTB decreases by 7.8%, and its parameter count by 34.8%. Finally, we propose a unified interpretation of both types of projection layers as computing similarities of input stimuli to learned prototypes for which we also propose a novel visualization technique highlighting the interpretability of Tversky projection layers. Our work offers a new paradigm for thinking about the similarity model implicit in modern deep learning, and designing neural networks that are interpretable under an established theory of psychological similarity.

Figures (18)

• Figure 1: (left) A single tversky projection layer with 11 learnable parameters that computes the xor function. The input is a 2 digit binary number encoded as $x^i \in \mathcal{R}^2$. The output $\hat{y}^i$ is such that $txor(x^i) \iff s(x^i,p^1) > s(x^i,p^0)$, where $s$ is the proposed similarity function. (middle) Input vectors $x^i$ along with features $f^i$, and prototypes $p^i$ modeling the $xor$ function. The dot products $x^1 \cdot f^1$, $x^2 \cdot f^0$, $p^1 \cdot f^0$ and $p^1 \cdot f^1$ are positive. (right) Feature vectors define a universe in which input vectors $x^0=\{\}$, $x^1=\{f^1\}$, $x^2=\{f^0\}$, $x^3=\{\}$, and learned prototype vectors $p^0=\{\}$, $p^1=\{ f^0, f^1\}$, are represented as set of features. By convention, an object (input or prototype) contains a feature if its dot product with the feature vector is positive. By Tversky's contrast model of similarity, $s(x^3,p^0) > s(x^3,p^1)$, and $s(x^2,p^1) > s(x^2,p^0)$ therefore $txor([1, 1])=0$ and $txor([1, 0])=1$. Similarly for other inputs, $txor([0, 0])=0$ and $txor([0, 1])=1$.
• Figure 2: Visualization of prototypes using our input domain projection parameter specification method. Top: 3 examples for each class. Bottom: 10 columns of the final linear projection layer of VisualMNISTNet, and 10 prototype vectors of the Tversky Projection Layer of TverskyMNISTNet. Both models achieve 99% accuracy on the validation set. Handwritten digit features are more perceptible to humans in Tversky prototypes and features compared to the linear projection prototypes.
• Figure 3: MNIST digits sorted from low to high salience (Equation \ref{['eq:saliency']}). Like humans, the underlying Tversky Neural Network perceives stimuli exhibiting more goodness of form as more salient.
• Figure 4: A tversky layer that adds 2 binary digits. Inputs are encoded in $\mathcal{R}^3$ as $x^0=[0,0,0]$, $x^1=[0,1,0]$, $x^2=[1,0,0]$ and $x^3=[1,1,1]$. The result of the addition corresponds to the prototype to which the input is most similar. Compared to the txor model, this model employs one additional dimention to enable the introduction of a feature $f^2=[0,0,1]$ only shared by $x^3$ and $p^2=[1,1,0.5]$. In this configuration, we have $tadd([0, 0])=0$, $tadd([0, 1])=tadd([1,0])=1$ and $tadd([1,1])=2$
• Figure 5: Illustration of our proposed data-domain visualization of projection parameters. Left: Classical deep neural network with a deep projection layer (Layer n) and its projection parameters illustrated as blue vectors. Right: Our proposed method. Projection parameters are specified as tensors of the same shape as the input data, and forwarded through the neural network up to layer n-1. The obtained vectors are used as projection parameters in Layer n. Using this method, the effective neural network parameter count is larger. However, this method enables visualizing the projection parameters in the input domain.