Janossy Pooling: Learning Deep Permutation-Invariant Functions for Variable-Size Inputs

TL;DR

The paper addresses learning permutation-invariant functions over variable-size inputs using Janossy pooling, which averages a permutation-sensitive function over all input orderings. It unifies existing approaches and studies two main strategies: exact $k$-ary Janossy pooling with a potential neural mapper and random-permutation (pi-SGD) approximations that modify the loss relationship. It highlights the trade-offs between model capacity and tractability, and shows that random-permutation methods can yield strongest empirical performance, while leaving theoretical gaps in loss equivalence and convergence. The work points to future extensions, including applications to graphs and non-Poisson point processes, and deeper theoretical understanding of the loss functions involved.

Abstract

We consider a simple and overarching representation for permutation-invariant functions of sequences (or multiset functions). Our approach, which we call Janossy pooling, expresses a permutation-invariant function as the average of a permutation-sensitive function applied to all reorderings of the input sequence. This allows us to leverage the rich and mature literature on permutation-sensitive functions to construct novel and flexible permutation-invariant functions. If carried out naively, Janossy pooling can be computationally prohibitive. To allow computational tractability, we consider three kinds of approximations: canonical orderings of sequences, functions with $k$-order interactions, and stochastic optimization algorithms with random permutations. Our framework unifies a variety of existing work in the literature, and suggests possible modeling and algorithmic extensions. We explore a few in our experiments, which demonstrate improved performance over current state-of-the-art methods.