Stochastic Adversarial Networks for Multi-Domain Text Classification

TL;DR

The paper tackles multi-domain text classification by eliminating the need for a growing set of domain-specific extractors. It introduces Stochastic Adversarial Network (SAN), which models domain-specific weights as a Gaussian distribution $\mathcal{N}(\mu,\Sigma)$ and samples with the reparameterization trick to keep parameter counts constant as new domains are added. To stabilize adversarial training and improve discriminability, SAN incorporates Domain Label Smoothing and Robust Pseudo-Label Regularization, including a Gaussian–uniform mixture model for pseudo-label reliability. Empirical results on two MDTC benchmarks show SAN achieves competitive accuracy with substantially reduced parameters and faster convergence, with noted improvements in efficiency and transfer to unseen domains. The work highlights a scalable, parameter-efficient approach to MDTC, though it also points to areas for improvement in pseudo-label quality to further enhance performance.

Abstract

Adversarial training has been instrumental in advancing multi-domain text classification (MDTC). Traditionally, MDTC methods employ a shared-private paradigm, with a shared feature extractor for domain-invariant knowledge and individual private feature extractors for domain-specific knowledge. Despite achieving state-of-the-art results, these methods grapple with the escalating model parameters due to the continuous addition of new domains. To address this challenge, we introduce the Stochastic Adversarial Network (SAN), which innovatively models the parameters of the domain-specific feature extractor as a multivariate Gaussian distribution, as opposed to a traditional weight vector. This design allows for the generation of numerous domain-specific feature extractors without a substantial increase in model parameters, maintaining the model's size on par with that of a single domain-specific extractor. Furthermore, our approach integrates domain label smoothing and robust pseudo-label regularization to fortify the stability of adversarial training and to refine feature discriminability, respectively. The performance of our SAN, evaluated on two leading MDTC benchmarks, demonstrates its competitive edge against the current state-of-the-art methodologies. The code is available at https://github.com/wangxu0820/SAN.

Figures (8)

• Figure 1: The architecture of the SAN method.
• Figure 2: An example of DLS with three domains on the FDU-MTL dataset.
• Figure 3: The distribution of the flattened $\Sigma$ values for initialization on the Amazon Review dataset.
• Figure 4: The distribution of the flattened $\Sigma$ values after convergence of SAN on the Amazon Review dataset.
• Figure 5: Parameter sensitivity analysis about $\lambda$ on Amazon review dataset.