Refining the Information Bottleneck via Adversarial Information Separation

TL;DR

This paper tackles generalization under data scarcity in scientific domains by separating task-relevant information from confounding noise without explicit supervision. It introduces AdverISF, a dual-branch framework with adversarial information separation and a multi-layer architecture that recycles noise to recover subtle, predictive features otherwise lost under uniform compression. The approach uses a self-supervised adversarial mechanism based on joint vs marginal distributions and WGAN-GP training, plus a cascaded design to progressively refine representations. Across synthetic benchmarks and a real-world composite cement design task, AdverISF achieves superior data-scarce performance and better out-of-distribution generalization than strong baselines, highlighting its practical impact for material science and related domains with limited data.

Abstract

Generalizing from limited data is particularly critical for models in domains such as material science, where task-relevant features in experimental datasets are often heavily confounded by measurement noise and experimental artifacts. Standard regularization techniques fail to precisely separate meaningful features from noise, while existing adversarial adaptation methods are limited by their reliance on explicit separation labels. To address this challenge, we propose the Adversarial Information Separation Framework (AdverISF), which isolates task-relevant features from noise without requiring explicit supervision. AdverISF introduces a self-supervised adversarial mechanism to enforce statistical independence between task-relevant features and noise representations. It further employs a multi-layer separation architecture that progressively recycles noise information across feature hierarchies to recover features inadvertently discarded as noise, thereby enabling finer-grained feature extraction. Extensive experiments demonstrate that AdverISF outperforms state-of-the-art methods in data-scarce scenarios. In addition, evaluations on real-world material design tasks show that it achieves superior generalization performance.

Figures (8)

• Figure 1: (a) A tight bottleneck retains dominant features (blue) but discards subtle features (green). (b) A loose bottleneck captures subtle features but leaks significant noise (gray). (c) Ours employs a layer-wise mechanism to recycle subtle features that would otherwise be erroneously discarded, effectively preserving meaningful information while filtering out noise.
• Figure 2: Schematic of the AdverISF. The architecture adopts a Multi-layer Separation strategy ($l=1$ to $L$) for progressive feature refinement. At each level, the Adversarial Information Separation Block utilizes dual encoders to decompose the input into task-relevant features ($z_{task}$) and noise representations ($z_{noise}$). Reparameterization (RT) is applied to enable differentiable sampling from the latent distributions. A central Adversarial Separation Mechanism enforces statistical independence between these latent codes via a minimax game, where a discriminator $D$ distinguishes between paired (joint) and shuffled (marginal) distributions. The noise output from the previous layer serves as input to the subsequent layer to capture subtle features $z_{subtle}$ (represented by $z_{task}^L$ in Layer $L$).
• Figure 3: Ablation study results. We compare the full AdverISF model (A0 (Full Model)) against three variants: A1 (w/o Multi-layer), A2 (w/o Variational), and A3 (w/o Adversarial). (a) Performance on the Concrete dataset across varying training set sizes ($N$). (b) Performance on the Synthetic dataset across varying training data ratios.
• Figure 4: Hyperparameter analysis on the Synthetic dataset. (a) Heatmaps showing $R^2$ stability across varying latent dimensions for layer 1 and layer 2. (b) Impact of KL divergence weights ($\beta$) on task and noise encoders. (c) Model performance robustness with respect to the adversarial loss weight $\lambda_{adv}$.
• Figure 5: Hyperparameter sensitivity analysis on Ratio 0.3 using Joint Training. The heatmap and curves illustrate performance stability across latent dimensions and weights.