Adapting, Fast and Slow: Transportable Circuits for Few-Shot Learning

TL;DR

The paper tackles domain generalization under distribution shifts by introducing a causal framework that uses qualitative graphs and a discrepancies oracle to enable zero-shot and few-shot transfer. It defines module-transportability and circuit-transportability, and proposes Circuit-TR and circuit-AD to transport and adapt modular predictors across domains. Theoretical results connect few-shot learnability to circuit size, with high-probability error-rate bounds that depend on transported components, while a gradient-based architecture provides a scalable approximation to the symbolic algorithms. Empirical validation on controlled simulations demonstrates fast adaptation when circuit-transportability holds and clarifies the trade-offs when it does not, offering a practical pathway for combining causal structure with scalable adaptation. The work advances causal transfer learning by formalizing transportable circuits and bridging theory with a tractable pretraining-finetuning paradigm for real-world domain adaptation tasks.

Abstract

Generalization across the domains is not possible without asserting a structure that constrains the unseen target domain w.r.t. the source domain. Building on causal transportability theory, we design an algorithm for zero-shot compositional generalization which relies on access to qualitative domain knowledge in form of a causal graph for intra-domain structure and discrepancies oracle for inter-domain mechanism sharing. \textit{Circuit-TR} learns a collection of modules (i.e., local predictors) from the source data, and transport/compose them to obtain a circuit for prediction in the target domain if the causal structure licenses. Furthermore, circuit transportability enables us to design a supervised domain adaptation scheme that operates without access to an explicit causal structure, and instead uses limited target data. Our theoretical results characterize classes of few-shot learnable tasks in terms of graphical circuit transportability criteria, and connects few-shot generalizability with the established notion of circuit size complexity; controlled simulations corroborate our theoretical results.

Figures (14)

• Figure 1: Causal diagrams corresponding to \ref{['ex:structure-informed-multi']}. Color-coded edges show parents of $Y$ in each domain: blue for $\mathcal{M}^1$, orange for $\mathcal{M}^*$. Single edges are the first parent and double edges are the second parent.
• Figure 2: Causal graphs corresponding to \ref{['ex:beyond-module-TR']}. In the sources, the mechanisms determining $Y$ are three noisy operators. The target SCM implements a GCD algorithm via the same three operators through a repeated structure. The query of interest is $P^*(v_{3|\mathcal{V}|}\mid v_1,v_2)$
• Figure 3: Implicit causal discovery in pretraining.
• Figure 4: The performance of our method which is based on structure agnostic domain adaptation, in comparison with the baselines that train jointly on source and target data, either by discarding the domain indices (ERM-pool) or by keeping them (ERM-joint).
• Figure 5: Selection diagram $\mathcal{G}^{\Delta}$

Theorems & Definitions (26)

• Definition 1.1
• Example 2.1: Motivating example
• Definition 2.2: Domain discrepancy sets
• Proposition 2.3: Module-TR
• Example 2.4: Beyond module-transportability
• Definition 2.5: Discrepancy oracle
• Definition 2.6
• Theorem 2.7: Circuit-TR error rate
• Example 3.1
• Theorem 3.2: Circuit-adaptation rate