Approximation-Generalization Trade-offs under (Approximate) Group Equivariance
Mircea Petrache, Shubhendu Trivedi
TL;DR
This work provides the most general PAC-style bounds tying model symmetry to improved generalization, without requiring the transformation set to be finite or form a group, and extends the analysis to partial and approximate equivariance via the stabilizer $\mathsf{Stab}_\epsilon$. It formalizes the interplay between model equivariance error and data equivariance error, deriving conditions under which symmetry yields optimal generalization. The paper also examines mis-specification, showing how misaligned symmetries can harm performance, and introduces an approximation-error framework that balances symmetry strength with data symmetry through $\epsilon$ and $\lambda$. Together, these results offer a principled foundation for designing equivariant or approximately equivariant models and selecting the appropriate level of symmetry given data structure, with implications for GCNNs and other symmetry-aware architectures.
Abstract
The explicit incorporation of task-specific inductive biases through symmetry has emerged as a general design precept in the development of high-performance machine learning models. For example, group equivariant neural networks have demonstrated impressive performance across various domains and applications such as protein and drug design. A prevalent intuition about such models is that the integration of relevant symmetry results in enhanced generalization. Moreover, it is posited that when the data and/or the model may only exhibit $\textit{approximate}$ or $\textit{partial}$ symmetry, the optimal or best-performing model is one where the model symmetry aligns with the data symmetry. In this paper, we conduct a formal unified investigation of these intuitions. To begin, we present general quantitative bounds that demonstrate how models capturing task-specific symmetries lead to improved generalization. In fact, our results do not require the transformations to be finite or even form a group and can work with partial or approximate equivariance. Utilizing this quantification, we examine the more general question of model mis-specification i.e. when the model symmetries don't align with the data symmetries. We establish, for a given symmetry group, a quantitative comparison between the approximate/partial equivariance of the model and that of the data distribution, precisely connecting model equivariance error and data equivariance error. Our result delineates conditions under which the model equivariance error is optimal, thereby yielding the best-performing model for the given task and data. Our results are the most general results of their type in the literature.