Discrete Latent Structure in Neural Networks

TL;DR

The paper surveys learning with discrete latent structure in neural networks, addressing how to induce and train structured, discrete representations that capture compositional structures such as trees and matchings. It organizes approaches into three broad families: continuous relaxations that yield differentiable proxies for discrete decisions, surrogate gradients that keep forward discreteness but replace backward signals, and probabilistic latent-variable methods that perform explicit marginalization or sampling. Each family is connected through common building blocks—maximization, marginalization, and sampling—and the authors map out the trade-offs in terms of tractability, bias, and gradient quality. The work highlights key techniques such as softmax- and Sinkhorn-based relaxations, SparseMAP, SPIGOT, Gumbel-Softmax, and REINFORCE, and discusses how these methods apply to structure prediction tasks and latent-variable learning. It also provides a practical roadmap with libraries to implement these methods, guiding researchers in choosing approaches aligned with their structural constraints and computational budgets.

Abstract

Many types of data from fields including natural language processing, computer vision, and bioinformatics, are well represented by discrete, compositional structures such as trees, sequences, or matchings. Latent structure models are a powerful tool for learning to extract such representations, offering a way to incorporate structural bias, discover insight about the data, and interpret decisions. However, effective training is challenging, as neural networks are typically designed for continuous computation. This text explores three broad strategies for learning with discrete latent structure: continuous relaxation, surrogate gradients, and probabilistic estimation. Our presentation relies on consistent notations for a wide range of models. As such, we reveal many new connections between latent structure learning strategies, showing how most consist of the same small set of fundamental building blocks, but use them differently, leading to substantially different applicability and properties.

Figures (12)

• Figure 1: Some example structures. Left: linear assignment (matching); center: dependency parse tree (directed arborescence); right: binary constituency parse tree (binary tree).
• Figure 2: Illustration of a selection of useful structures, along with their matrix representation.
• Figure 3: Shift-reduce probabilities for binary trees, with shift probability $.5$ (left) and $.8$ (right) modulo constraints.
• Figure 4: The graphical model of incremental prediction is a fully connected acyclic graph. The combinatorial explosion challenge is still present, but delegated to the incremental predictors $p(z_k \mid z_1, \ldots, z_{k-1})$.
• Figure 5: Samples drawn from a Needleman-Wunsch alignment model using a dynamic programming Forwards Filtering, Backwards Sampling strategy.

Theorems & Definitions (3)

• Remark
• Remark
• Definition 2.1