Distribution Learning Meets Graph Structure Sampling
Arnab Bhattacharyya, Sutanu Gayen, Philips George John, Sayantan Sen, N. V. Vinodchandran
TL;DR
The paper presents a novel online-learning reduction that connects PAC-learning of high-dimensional Bayes nets to efficient counting/sampling of graph structures. By applying Exponential Weighted Averaging (EWA) or Randomized Weighted Majority (RWM) forecasters on sequences of samples with log-loss, it derives regret-based bounds that translate into strong KL-divergence guarantees for learned distributions. The authors achieve near-optimal sample complexities for agnostic learning of indegree-bounded Bayes nets and provide polynomial-time algorithms for learning Bayes nets on known chordal skeletons and for tree-structured distributions, including a new tree-learning approach via the matrix-tree theorem. They also extend results to polytree and bounded-vertex-cover scenarios and discuss maximum likelihood adaptations, demonstrating the framework’s broad applicability to structured distribution learning and structure learning with efficient sampling. The work offers principled guarantees and scalable algorithms with potential impact on structure learning, causal inference, and high-dimensional graphical modeling.
Abstract
This work establishes a novel link between the problem of PAC-learning high-dimensional graphical models and the task of (efficient) counting and sampling of graph structures, using an online learning framework. We observe that if we apply the exponentially weighted average (EWA) or randomized weighted majority (RWM) forecasters on a sequence of samples from a distribution P using the log loss function, the average regret incurred by the forecaster's predictions can be used to bound the expected KL divergence between P and the predictions. Known regret bounds for EWA and RWM then yield new sample complexity bounds for learning Bayes nets. Moreover, these algorithms can be made computationally efficient for several interesting classes of Bayes nets. Specifically, we give a new sample-optimal and polynomial time learning algorithm with respect to trees of unknown structure and the first polynomial sample and time algorithm for learning with respect to Bayes nets over a given chordal skeleton.