Generalization Bounds for Equivariant Networks on Markov Data
Hui Li, Zhiguo Wang, Bohui Chen, Li Sheng
TL;DR
The paper addresses how Markovian dependencies interact with symmetry-driven inductive biases in neural networks by deriving a Markov-aware generalization bound for MLPs and extending it to equivariant networks. It introduces a novel adaptation of McDiarmid’s inequality that incorporates mixing time and an $L_2$-gap to quantify deviation from stationarity, yielding a bound that includes a term $C_n$ for initial–stationary discrepancy and a mixing-time dependent factor $\tau_{min}$. To handle equivariance, the authors employ the Peter–Weyl theorem to obtain a block-diagonal representation of weight matrices via irreducible representations, apply Schur’s lemma to constrain cross-block connections, and use Maurey sparsification to bound the covering number; these are combined with the Dudley entropy integral to bound the empirical Rademacher complexity of deep equivariant nets. The resulting bounds reveal that choosing smaller, non-isomorphic irreducible representations can tighten generalization, and the bounds provide practical guidance for architecture design under symmetry constraints. Experimental results on synthetic Markov torus data and rotated MNIST illustrate the impact of Markov structure and irreducible representation choices on generalization, validating the theoretical insights.
Abstract
Equivariant neural networks play a pivotal role in analyzing datasets with symmetry properties, particularly in complex data structures. However, integrating equivariance with Markov properties presents notable challenges due to the inherent dependencies within such data. Previous research has primarily concentrated on establishing generalization bounds under the assumption of independently and identically distributed data, frequently neglecting the influence of Markov dependencies. In this study, we investigate the impact of Markov properties on generalization performance alongside the role of equivariance within this context. We begin by applying a new McDiarmid's inequality to derive a generalization bound for neural networks trained on Markov datasets, using Rademacher complexity as a central measure of model capacity. Subsequently, we utilize group theory to compute the covering number under equivariant constraints, enabling us to obtain an upper bound on the Rademacher complexity based on this covering number. This bound provides practical insights into selecting low-dimensional irreducible representations, enhancing generalization performance for fixed-width equivariant neural networks.