Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Approximation of relation functions and attention mechanisms

Awni Altabaa, John Lafferty

TL;DR

The paper proves that relation functions realized as inner products of neural encodings can be universally approximated in both symmetric and asymmetric settings, tying symmetric cases to RKHS kernels and asymmetric cases to RKBS kernels. It provides explicit neuron-count bounds linked to kernel spectrum decay and Barron-norm smoothness, and shows how attention mechanisms in Transformers can approximate retrieval or selection tasks via these inner-product relations, leveraging Debreu's representation theorem. The results offer a rigorous foundation for relational learning with neural nets, clarify the theoretical role of attention, and connect neural approximations to classical kernel theory and economic representations. Overall, the work bridges neural-network approximation theory, kernel methods, and attention mechanisms, offering principled bounds and mechanisms for efficient relational computation.

Abstract

Inner products of neural network feature maps arise in a wide variety of machine learning frameworks as a method of modeling relations between inputs. This work studies the approximation properties of inner products of neural networks. It is shown that the inner product of a multi-layer perceptron with itself is a universal approximator for symmetric positive-definite relation functions. In the case of asymmetric relation functions, it is shown that the inner product of two different multi-layer perceptrons is a universal approximator. In both cases, a bound is obtained on the number of neurons required to achieve a given accuracy of approximation. In the symmetric case, the function class can be identified with kernels of reproducing kernel Hilbert spaces, whereas in the asymmetric case the function class can be identified with kernels of reproducing kernel Banach spaces. Finally, these approximation results are applied to analyzing the attention mechanism underlying Transformers, showing that any retrieval mechanism defined by an abstract preorder can be approximated by attention through its inner product relations. This result uses the Debreu representation theorem in economics to represent preference relations in terms of utility functions.

Approximation of relation functions and attention mechanisms

TL;DR

The paper proves that relation functions realized as inner products of neural encodings can be universally approximated in both symmetric and asymmetric settings, tying symmetric cases to RKHS kernels and asymmetric cases to RKBS kernels. It provides explicit neuron-count bounds linked to kernel spectrum decay and Barron-norm smoothness, and shows how attention mechanisms in Transformers can approximate retrieval or selection tasks via these inner-product relations, leveraging Debreu's representation theorem. The results offer a rigorous foundation for relational learning with neural nets, clarify the theoretical role of attention, and connect neural approximations to classical kernel theory and economic representations. Overall, the work bridges neural-network approximation theory, kernel methods, and attention mechanisms, offering principled bounds and mechanisms for efficient relational computation.

Abstract

Inner products of neural network feature maps arise in a wide variety of machine learning frameworks as a method of modeling relations between inputs. This work studies the approximation properties of inner products of neural networks. It is shown that the inner product of a multi-layer perceptron with itself is a universal approximator for symmetric positive-definite relation functions. In the case of asymmetric relation functions, it is shown that the inner product of two different multi-layer perceptrons is a universal approximator. In both cases, a bound is obtained on the number of neurons required to achieve a given accuracy of approximation. In the symmetric case, the function class can be identified with kernels of reproducing kernel Hilbert spaces, whereas in the asymmetric case the function class can be identified with kernels of reproducing kernel Banach spaces. Finally, these approximation results are applied to analyzing the attention mechanism underlying Transformers, showing that any retrieval mechanism defined by an abstract preorder can be approximated by attention through its inner product relations. This result uses the Debreu representation theorem in economics to represent preference relations in terms of utility functions.
Paper Structure (8 sections, 8 theorems, 72 equations)

This paper contains 8 sections, 8 theorems, 72 equations.

Table of Contents

  1. Introduction
  2. Function class of symmetric inner product relations
  3. Function class of asymmetric inner product relations
  4. Connection to reproducing kernel Hilbert and Banach spaces
  5. Background on reproducing kernel Banach spaces
  6. Asymmetric inner products of neural networks model kernels of reproducing kernel Banach spaces
  7. Application: Analysis of Attention
  8. Discussion

Key Result

theorem 1

Suppose the data lies in a compact Euclidean space $\mathcal{X}$. Let $\calF$ be a function space from $\calX$ to $\reals$, and $\nnorm{\cdot}_\calF$ a norm on functions from $\calX$ to $\reals$. Let $r: \calX \times \calX$ be a symmetric positive-definite kernel satisfying ass:sym_pd_ker_specturm_d Moreover, the complexity of the neural network required is bounded by

Theorems & Definitions (22)

  • theorem 1: Function class of symmetric inner product relational neural networks
  • proof
  • remark 1
  • corollary 1
  • corollary 2
  • lemma 1
  • theorem 2
  • proof
  • remark 2
  • remark 3
  • ...and 12 more