Generalist++: A Meta-learning Framework for Mitigating Trade-off in Adversarial Training

TL;DR

This paper tackles two core issues in adversarial training: the natural-robustness trade-off and robustness across different norm constraints. It introduces Generalist, a meta-learning framework that partitions AT into task-aware base learners and a global aggregator that learns by alternating between independent specialization and periodic parameter exchange. Theoretical results provide a regret-based generalization bound and an $\,\\epsilon$-stability bound, while extensive experiments on CIFAR, ImageNet, and OOD benchmarks show that Generalist reduces generalization error and alleviates both trade-offs, often outperforming strong baselines with the same model size. Overall, Generalist demonstrates that decoupled, task-aware learning with principled aggregation can yield more robust and transferable classifiers in practical settings.

Abstract

Despite the rapid progress of neural networks, they remain highly vulnerable to adversarial examples, for which adversarial training (AT) is currently the most effective defense. While AT has been extensively studied, its practical applications expose two major limitations: natural accuracy tends to degrade significantly compared with standard training, and robustness does not transfer well across attacks crafted under different norm constraints. Unlike prior works that attempt to address only one issue within a single network, we propose to partition the overall generalization goal into multiple sub-tasks, each assigned to a dedicated base learner. By specializing in its designated objective, each base learner quickly becomes an expert in its field. In the later stages of training, we interpolate their parameters to form a knowledgeable global learner, while periodically redistributing the global parameters back to the base learners to prevent their optimization trajectories from drifting too far from the shared target. We term this framework Generalist and introduce three variants tailored to different application scenarios. Both theoretical analysis and extensive experiments demonstrate that Generalist achieves lower generalization error and significantly alleviates the trade-off problems compared with baseline methods. Our results suggest that Generalist provides a promising step toward developing fully robust classifiers in the future.

Figures (12)

• Figure 1: Comparison with current variants of AT that aim at achieving a better trade-off. Note that the baseline for comparison is different in (a) and (b) because existing methods typically address one problem at a time. We compare Generalist against their respective areas of expertise. Results show that Generalist achieves strong performance when focusing on a single trade-off issue (see Generalist-D). Moreover, when addressing two issues simultaneously, Generalist outperforms existing baselines in both aspects (see Generalist-T). The improvement is notable since we only use the naive cross-entropy loss without increasing model size.
• Figure 2: Pipeline of the proposed Generalist. Multiple base learners are trained independently within their respective sub-tasks. A global learner periodically aggregates parameters from the base learners, integrates knowledge, and redistributes the updated parameters back for continued training.
• Figure 3:
• Figure 4: The impact of $\gamma_2$ to the performances of Generalist-T.
• Figure 5: The impact of $c$ to the performances of Generalist-T.

Theorems & Definitions (14)

• Definition 1
• Theorem 1
• Definition 2
• Theorem 2
• proof
• Proposition 1
• proof
• Remark 1
• Definition 3
• Theorem 3