Inferring Higher-Order Couplings with Neural Networks

TL;DR

This Letter introduces a method that maps RBMs onto generalized Potts models, enabling the systematic extraction of interactions up to arbitrary order, and establishes RBMs as a powerful and efficient tool for modeling higher-order structure for high-dimensional categorical data.

Abstract

Maximum entropy methods, rooted in the inverse Ising/Potts problem from statistical physics, are widely used to model pairwise interactions in complex systems across disciplines such as bioinformatics and neuroscience. While successful, these approaches often fail to capture higher-order interactions that are critical for understanding collective behavior. In contrast, modern machine learning methods can model such interactions, but their interpretability often comes at a prohibitive computational cost. Restricted Boltzmann Machines (RBMs) provide a computationally efficient alternative by encoding statistical correlations through hidden units in a bipartite architecture. In this work, we introduce a method that maps RBMs onto generalized Potts models, enabling the systematic extraction of interactions up to arbitrary order. Leveraging large-$N$ approximations, made tractable by the RBM's structure, we extract effective many-body couplings with minimal computational effort. We further propose a robust framework for recovering higher-order interactions in more complex generative models, and introduce a simple gauge-fixing scheme for the effective Potts representation. Validation on synthetic data demonstrates accurate recovery of two- and three-body interactions. Applied to protein sequence data, our method reconstructs contact maps with high fidelity and outperforms state-of-the-art inverse Potts models. These results establish RBMs as a powerful and efficient tool for modeling higher-order structure in high-dimensional categorical data.

Figures (10)

• Figure 1: (a) Potts-Bernoulli RBM architecture with binary hidden units $h_a$ and $q$-state visible units $v_i$. (b) Time evolution of effective couplings inferred by an RBM trained on equilibrium configurations from the spin-1 Blume-Capel model (Eq. \ref{['blume-capel_hamiltonian']}, $\beta J^{(2)} = \beta J^{(3)} = 0.2$). Solid lines show mean inferred values in the zero-sum gauge for $n=2$ and $n=3$ interactions, averaged separately over coupled and uncoupled terms; shaded areas indicate standard deviations among pairs/triplets. Dashed lines mark ground-truth couplings. Training hyperparameters are noted in the title: PCD-$k$ (persistent contrastive divergence with $k$ steps), ${N_\mathrm{h}}$ (number of hidden units), and $\gamma$ (learning rate). (c) Inferred pairwise coupling matrix (below diagonal) vs. ground truth (above diagonal). (d) Normalized signal $\hat{R}^{(2)}_{i_1,i_2}$ indicating deviations from a purely pairwise model, compared to the true three-body interactions, displayed as nonzero entries across the corresponding pairs in a $2 \times 2$ grid.
• Figure 2: RBM-based contact prediction on the Response Regulator Receiver Domain (Pfam: PF00072). We evaluate the RBM on the PF00072 MSA for residue-residue contact prediction, comparing its performance to DCA-based methods (plmDCA ekeberg2013improvedekeberg2014fast and adabmDCA rosset2025adabmdca) using public implementations. (a1): RBM log-likelihood (LL) (solid: train, dashed: test) and final adamDCA LL (horizontal line) during training. (a2): AUC for ROC (solid) and PPV (dashed) during training; horizontal lines indicate plmDCA and adabmDCA. (a3): Frobenius norms of effective $n$-body interactions ($n\!=\!1,2,3$) vs. training time; RBM surpasses baselines as 3-body terms emerge. (b): Comparison of 3-point statistics from RBM- and adabmDCA-generated samples vs. training data; red: test vs. train. Legend: Pearson correlation ($r$), slope of linear fit ($m$). (c1): ROC curves; inset (c2): PPV vs. number of top-ranked predictions (being 1 for the top 50 for all methods). (d): Contact maps, experimental (light gray), RBM (upper triangle), plmDCA (lower); red: true positives, green: false positives.
• Figure 3: Diagram of the 3-visible hidden nodes RBM we use to derive the exact mapping in the zero-sum Gauge.
• Figure 4: Sample distribution of the sum of the weights randomly drawn from the (a) 235-th, (b) 22-th, (c) 175-th, and (d) 15-th columns of the weight matrix of the RBM trained with the modified Blume-Capel model samples (at $t=5 \cdot 10^6$). The dashed black line is $X_a$ under the single Gaussian approximation given in \ref{['eq:CLT']}, while the solid red line is the approximation using a Gaussian mixture \ref{['CLT_assumption_2']} with $\tau=2.5$. The inset shows a barplot with the $\| \boldsymbol{\hat{W}}_{ia} \|$ and the $\hat{R}_{i,a}^{(1)}$ computed for one hidden $a$ and visible $i$ node connection. These quantities are rescaled to 0.0 and 1.0.
• Figure 5: 2-Body couplings extracted with a simple Gaussian and a Gaussian mixture approximation from the RBM trained with Blume-Capel model samples (at $t=5 \cdot 10^6$).

Theorems & Definitions (4)

• Theorem 1
• proof
• Theorem 2
• proof