Game-theoretic distributed learning of generative models for heterogeneous data collections

TL;DR

This work tackles heterogeneity in distributed learning by reframing local models as players in a cooperative game that exchanges synthetic data instead of raw parameters. It develops a rigorous framework with a Nash equilibrium guarantee for exponential-family local models and proves convergence of the proposed learning dynamics, including a dual formulation based on entropy. The approach naturally extends to semi-supervised and multimodal settings, enabling local models to operate on different probability spaces while still benefiting from cross-model information. Empirical results on MNIST, Fashion MNIST, and PolyMNIST demonstrate improved cross-domain generation and classification performance, validating the practicality and scalability of synthetic-data–based collaboration for heterogeneous data collections.

Abstract

One of the main challenges in distributed learning arises from the difficulty of handling heterogeneous local models and data. In light of the recent success of generative models, we propose to meet this challenge by building on the idea of exchanging synthetic data instead of sharing model parameters. Local models can then be treated as ``black boxes'' with the ability to learn their parameters from data and to generate data according to these parameters. Moreover, if the local models admit semi-supervised learning, we can extend the approach by enabling local models on different probability spaces. This allows to handle heterogeneous data with different modalities. We formulate the learning of the local models as a cooperative game starting from the principles of game theory. We prove the existence of a unique Nash equilibrium for exponential family local models and show that the proposed learning approach converges to this equilibrium. We demonstrate the advantages of our approach on standard benchmark vision datasets for image classification and conditional generation.

Figures (3)

• Figure 1: MNIST experiment. Top: images generated by the learned models (we show probabilities $p(x\,\vert\, z_0,z_1)$ instead of sampled binary images for better visibility), bottom: dependencies of the obtained FID scores on $\alpha$ (see text for explanation).
• Figure 2: Fashion MNIST experiments. Classification accuracies of the trained models in dependence on the number of training examples. Four panels correspond to four models with different expressive power (see text for explanations).
• Figure 3: PolyMNIST experiments. Top: original images, middle: generated images, rows and columns correspond to styles and digits, respectively. Bottom: classification accuracies per style for the considered models, "avg" -- averaged over styles.

Theorems & Definitions (2)

• Theorem 1
• proof