Statistical Learning Theory for Distributional Classification
Christian Fiedler
TL;DR
This work addresses the problem of learning classifiers when inputs are probability distributions observed only through samples, using kernel-embedding techniques to map distributions into a Hilbert space and applying SVMs on the embeddings. It establishes a two-stage sampling oracle inequality for SVMs, proves universal consistency, and derives learning-rate results under margin/noise-type assumptions, with a focused treatment of Gaussian kernels and hinge loss. A key technical advancement is a novel feature-space construction for Gaussian kernels on Hilbert spaces via Gaussian measures and the white-noise mapping, enabling tractable analysis in infinite dimensions and yielding explicit approximation-error bounds. The results provide rigorous, practically relevant guarantees for distributional classification, with applications in medical screening and causal learning, and lay groundwork for further refinements and extensions to other embeddings and learning settings.
Abstract
In supervised learning with distributional inputs in the two-stage sampling setup, relevant to applications like learning-based medical screening or causal learning, the inputs (which are probability distributions) are not accessible in the learning phase, but only samples thereof. This problem is particularly amenable to kernel-based learning methods, where the distributions or samples are first embedded into a Hilbert space, often using kernel mean embeddings (KMEs), and then a standard kernel method like Support Vector Machines (SVMs) is applied, using a kernel defined on the embedding Hilbert space. In this work, we contribute to the theoretical analysis of this latter approach, with a particular focus on classification with distributional inputs using SVMs. We establish a new oracle inequality and derive consistency and learning rate results. Furthermore, for SVMs using the hinge loss and Gaussian kernels, we formulate a novel variant of an established noise assumption from the binary classification literature, under which we can establish learning rates. Finally, some of our technical tools like a new feature space for Gaussian kernels on Hilbert spaces are of independent interest.