Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Topology-aware Robust Optimization for Out-of-distribution Generalization

Fengchun Qiao, Xi Peng

TL;DR

This work addresses OOD generalization by introducing topology-aware robust optimization (TRO), which explicitly leverages the topology among data distributions. TRO operates in two phases: topology learning (constructing a distributional graph from physical priors or data-driven diffusion-based measures) and learning on topology (constrained robust optimization over mixtures of training groups guided by a topology-derived prior). The authors prove fast convergence for convex losses and establish non-convex convergence and generalization bounds that incorporate the topology, and they demonstrate state-of-the-art results across classification, regression, and semantic segmentation with both physical and data-driven topologies. The data-driven topology, in particular, aligns with domain knowledge and enhances explainability, suggesting TRO as a practical and theoretically sound framework for reliable OOD resilience in real-world settings.

Abstract

Out-of-distribution (OOD) generalization is a challenging machine learning problem yet highly desirable in many high-stake applications. Existing methods suffer from overly pessimistic modeling with low generalization confidence. As generalizing to arbitrary test distributions is impossible, we hypothesize that further structure on the topology of distributions is crucial in developing strong OOD resilience. To this end, we propose topology-aware robust optimization (TRO) that seamlessly integrates distributional topology in a principled optimization framework. More specifically, TRO solves two optimization objectives: (1) Topology Learning which explores data manifold to uncover the distributional topology; (2) Learning on Topology which exploits the topology to constrain robust optimization for tightly-bounded generalization risks. We theoretically demonstrate the effectiveness of our approach and empirically show that it significantly outperforms the state of the arts in a wide range of tasks including classification, regression, and semantic segmentation. Moreover, we empirically find the data-driven distributional topology is consistent with domain knowledge, enhancing the explainability of our approach.

Topology-aware Robust Optimization for Out-of-distribution Generalization

TL;DR

This work addresses OOD generalization by introducing topology-aware robust optimization (TRO), which explicitly leverages the topology among data distributions. TRO operates in two phases: topology learning (constructing a distributional graph from physical priors or data-driven diffusion-based measures) and learning on topology (constrained robust optimization over mixtures of training groups guided by a topology-derived prior). The authors prove fast convergence for convex losses and establish non-convex convergence and generalization bounds that incorporate the topology, and they demonstrate state-of-the-art results across classification, regression, and semantic segmentation with both physical and data-driven topologies. The data-driven topology, in particular, aligns with domain knowledge and enhances explainability, suggesting TRO as a practical and theoretically sound framework for reliable OOD resilience in real-world settings.

Abstract

Out-of-distribution (OOD) generalization is a challenging machine learning problem yet highly desirable in many high-stake applications. Existing methods suffer from overly pessimistic modeling with low generalization confidence. As generalizing to arbitrary test distributions is impossible, we hypothesize that further structure on the topology of distributions is crucial in developing strong OOD resilience. To this end, we propose topology-aware robust optimization (TRO) that seamlessly integrates distributional topology in a principled optimization framework. More specifically, TRO solves two optimization objectives: (1) Topology Learning which explores data manifold to uncover the distributional topology; (2) Learning on Topology which exploits the topology to constrain robust optimization for tightly-bounded generalization risks. We theoretically demonstrate the effectiveness of our approach and empirically show that it significantly outperforms the state of the arts in a wide range of tasks including classification, regression, and semantic segmentation. Moreover, we empirically find the data-driven distributional topology is consistent with domain knowledge, enhancing the explainability of our approach.
Paper Structure (19 sections, 25 equations, 14 figures, 7 tables, 1 algorithm)

This paper contains 19 sections, 25 equations, 14 figures, 7 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Problem Formulation and Preliminary Works
  3. Topology-aware robust optimization
  4. Topology learning: Explore the distributional topology
  5. Learning on topology: Exploit topology for robust optimization
  6. Theoretical Analysis
  7. Convergence Analysis
  8. Generalization Bounds
  9. Experiments
  10. Classification
  11. Regression
  12. Semantic Segmentation
  13. Conclusion
  14. Appendix
  15. Related Work
  16. ...and 4 more sections

Figures (14)

  • Figure 1: Overview of topology-aware distributionally robust optimization (TRO).
  • Figure 2: Illustration of data groups in (a) DG-15 and (b) DG-60 datasets.
  • Figure 3: Group importance of DRO and TRO on DG-15. DRO assigns the highest weight to the worst-case group "1" which is the furthest group to the test groups while TRO focuses on the influential groups "2", "5", and "6", which are truly connected to test groups.
  • Figure 3: MSE on TPT-48. Ignoring either the worst-case (IW-ERM) or influential (DRO) groups would yield compromised performance.
  • Figure 4: Left: Generalization task of North $\rightarrow$ South on TPT-48. Middle: Group centrality of physical-based topology. Right: Group centrality of data-driven topology. "PA" is identified by TRO as the influential group in physical-based topology; "NY", "PA", and "MA" are identified by TRO as influential groups in data-driven topology. Data topology yields lower MSE than physical topology.
  • ...and 9 more figures