A Theory of Machine Learning

TL;DR

The paper interrogates three foundational theories of machine learning—two epistemic (possible worlds and recognition) and one behavioral (operation theory)—and proposes a new computation-based theory in which learning occurs only when a machine successfully computes a target function. It formalizes a success criterion and demonstrates that learning true probabilities is not simply a correct calculation, almost-sure convergence, or calibration, using a sequence of theorems (3–6) and a necessary condition. By contrasting theoretical limits with practical implications, it shows that learning true probabilities can be feasible when probabilities are directly observable (as in NLP with an ideal corpus W*), but unlearnable in model-based macroeconomic settings (e.g., r-star), supported by Theorems 8–10. Collectively, the work clarifies when and how true probabilistic knowledge can be learned and highlights the trade-offs between epistemic and operational notions of learning for algorithm design and analysis.

Abstract

We critically review three major theories of machine learning and provide a new theory according to which machines learn a function when the machines successfully compute it. We show that this theory challenges common assumptions in the statistical and the computational learning theories, for it implies that learning true probabilities is equivalent neither to obtaining a correct calculation of the true probabilities nor to obtaining an almost-sure convergence to them. We also briefly discuss some case studies from natural language processing and macroeconomics from the perspective of the new theory.