A Theory of the Mechanics of Information: Generalization Through Measurement of Uncertainty (Learning is Measuring)
Christopher J. Hazard, Michael Resnick, Jacob Beel, Jack Xia, Cade Mack, Dominic Glennie, Matthew Fulp, David Maze, Andrew Bassett, Martin Koistinen
TL;DR
The paper proposes a model-free, information-theoretic framework for learning and inference built on surprisal, aiming to replace distribution-modeling approaches with a data-centric, traceable method. By treating uncertainty as a distance and deriving discriminative/generative inferences, the authors unify tasks such as causal discovery, anomaly detection, time-series forecasting, and supervised learning under a single, interpretable paradigm. Key contributions include formal definitions of deviations, feature influence, and conviction-based diagnostics; a nearest-neighbors-like inference engine; and a scalable implementation (Howso Engine/Amalgam) with data compression, privacy-aware synthesis, and reinforcement-learning connections. Empirically, the approach demonstrates competitive performance across diverse tasks, robustness to missing data and adversarial conditions, and promising data-efficient scalability via hierarchical sharding and ablation. Collectively, this work foregrounds a physics-of-information viewpoint as a viable, human-understandable alternative path to neural networks for scalable AI.
Abstract
Traditional machine learning relies on explicit models and domain assumptions, limiting flexibility and interpretability. We introduce a model-free framework using surprisal (information theoretic uncertainty) to directly analyze and perform inferences from raw data, eliminating distribution modeling, reducing bias, and enabling efficient updates including direct edits and deletion of training data. By quantifying relevance through uncertainty, the approach enables generalizable inference across tasks including generative inference, causal discovery, anomaly detection, and time series forecasting. It emphasizes traceability, interpretability, and data-driven decision making, offering a unified, human-understandable framework for machine learning, and achieves at or near state-of-the-art performance across most common machine learning tasks. The mathematical foundations create a ``physics'' of information, which enable these techniques to apply effectively to a wide variety of complex data types, including missing data. Empirical results indicate that this may be a viable alternative path to neural networks with regard to scalable machine learning and artificial intelligence that can maintain human understandability of the underlying mechanics.