Le Cam Distortion: A Decision-Theoretic Framework for Robust Transfer Learning

TL;DR

This work reframes transfer learning under distribution shift from symmetric invariance to directional simulability grounded in Le Cam's theory of statistical experiments. It defines Le Cam Distortion via deficiency distance and proves a suite of results (Transfer, Hinge, Directionality, Experiment Dominance) that guarantee risk control when a source experiment can simulate a target with bounded error. The authors introduce a practical method to learn degradation simulators (K) using representational encoders and MMD proxies, demonstrating safe transfer with preserved source utility across Gaussian shifts, CIFAR-10, RL control, and discrete HLA genomics. Across continuous and discrete domains, Le Cam Distortion yields safer, more reliable transfer in safety-critical settings, at the cost of potentially reduced target performance in exchange for robust source reliability. This framework unifies classical statistics with modern ML concepts (domain adaptation, RL, generative modeling) and offers principled design principles for when to prioritize safety over aggressive transfer.

Abstract

Distribution shift is the defining challenge of real-world machine learning. The dominant paradigm--Unsupervised Domain Adaptation (UDA)--enforces feature invariance, aligning source and target representations via symmetric divergence minimization [Ganin et al., 2016]. We demonstrate that this approach is fundamentally flawed: when domains are unequally informative (e.g., high-quality vs degraded sensors), strict invariance necessitates information destruction, causing "negative transfer" that can be catastrophic in safety-critical applications [Wang et al., 2019]. We propose a decision-theoretic framework grounded in Le Cam's theory of statistical experiments [Le Cam, 1986], using constructive approximations to replace symmetric invariance with directional simulability. We introduce Le Cam Distortion, quantified by the Deficiency Distance $δ(E_1, E_2)$, as a rigorous upper bound for transfer risk conditional on simulability. Our framework enables transfer without source degradation by learning a kernel that simulates the target from the source. Across five experiments (genomics, vision, reinforcement learning), Le Cam Distortion achieves: (1) near-perfect frequency estimation in HLA genomics (correlation $r=0.999$, matching classical methods), (2) zero source utility loss in CIFAR-10 image classification (81.2% accuracy preserved vs 34.7% drop for CycleGAN), and (3) safe policy transfer in RL control where invariance-based methods suffer catastrophic collapse. Le Cam Distortion provides the first principled framework for risk-controlled transfer learning in domains where negative transfer is unacceptable: medical imaging, autonomous systems, and precision medicine.

Figures (4)

• Figure 1: RL Robustness Analysis under Observation Degradation.(A) Trajectories: State evolution in the noisy Target environment ($\sigma=1.0$). The Naive (Red) agent, trained on clean Source data, learns an aggressive control gain ($w \approx -1.0$) that is unstable under noise, leading to rapid divergence (oscillations growing unbounded). The Invariant (Green) agent, forced to minimize MMD between Clean Source and Noisy Target representations, learns to "ignore" the state signal to satisfy invariance, resulting in a zero-gain policy ($w \approx 0$) that fails to control the system (The Invariance Trap). The Le Cam (Blue) agent is trained on a simulated target environment ($P_{\text{sim}} \approx P_{\text{target}}$) generated by learning the degradation kernel. It learns a conservative gain ($w \approx -0.5$) that maintains stability and effective control, preventing catastrophe. (B) Quantitative Performance: Average returns (higher is better). Le Cam alignment achieves superior safety (Return: -25.3) compared to the Naive baseline (-50.3). The Invariant baseline incurs catastrophic costs (-1290.2), verifying that strict invariance destroys task-relevant information.
• Figure 2: 2D Control with Anisotropic Observation Noise. We visualize trajectories on the noisy Target domain where the Y-axis is severely degraded ($\sigma_Y = 2.0$) while X is reliable ($\sigma_X = 0.1$). Naive (Red Circles): Learns symmetric aggressive gains, failing to account for noise anisotropy; notice the wild vertical oscillations. Invariant (Green Squares): Collapses the Y-signal to satisfy MMD invariance, resulting in a policy that ignores Y-deviation and drifts vertically (unbounded loss). Le Cam (Blue Triangles): Learns the anisotropic noise profile ($\hat{\sigma} \approx [0.15, 2.09]$) via the directional simulator. It adapts by acting aggressively on X ($w_x \approx -1.0$) but conservatively on Y ($w_y \approx -0.34$), smoothly converging to the target (gold star). Le Cam achieves 29x better performance than Invariant, demonstrating that directional simulability enables dimension-specific robustness.
• Figure 3: The Invariance Trap in CIFAR-10 transfer. We plot Source (Clean) vs Target (Degraded) accuracy for all methods. CycleGAN (Orange Square) illustrates the trap: to align representations, it degrades Source accuracy from 81.0% to 46.3% (-34.7%), effectively "forgetting" high-frequency details to match the blurred Target. Le Cam (Green Triangle) prioritizes safety: it maintains Source accuracy at 81.2% (no drop), proving that deficiency minimization $\delta(S, T)$ enables safe reuse of Source utility. While Target transfer is partial (26.5%), it is strictly safer than the Invariant baseline. Source-Only (Blue Circle) serves as the reference bound.
• Figure 4: Test A2: Quantization Monotonicity. We verified that Le Cam Deficiency (simulability error) correctly identifies information loss. As quantization bin width $\Delta$ increases (x-axis), the estimated deficiency $\delta(S, T)$ (Blue line) increases monotonically. The theoretical risk inflation (Red dashed line) grows quadratically $O(\Delta^2)$, which is tightly tracked by the empirical deficiency. This confirms that deficiency is a valid proxy for potential downstream risk.

Theorems & Definitions (29)

• Definition 2.1: Statistical Experiment
• Remark 2.2: Parameter vs Environment
• Definition 2.3: Decision Problem
• Definition 2.4: Risk and Minimax Risk
• Definition 2.5: Markov Kernel
• Remark 2.6: Parameter-Independence Constraint
• Definition 2.7: Representation and Induced Experiment
• Definition 3.1: Total Variation Distance
• Definition 3.2: Le Cam Deficiency
• Remark 3.3: The Theory-Implementation Gap