Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

From Softmax to Sparsemax: A Sparse Model of Attention and Multi-Label Classification

André F. T. Martins, Ramón Fernandez Astudillo

TL;DR

This work introduces sparsemax, a differentiable activation that projects inputs onto the probability simplex to yield sparse output probabilities, enabling selective attention and multi-label predictions. It derives a closed-form threshold-based solution, the Jacobian for efficient backpropagation, and a convex sparsemax loss that mirrors logistic loss while promoting sparsity. The paper demonstrates competitive performance in multi-label classification and natural language inference tasks, with the sparsemax attention mechanism producing more compact and interpretable focus. Overall, sparsemax offers a principled alternative to softmax for scalable, sparse, and interpretable neural architectures with robust optimization properties.

Abstract

We propose sparsemax, a new activation function similar to the traditional softmax, but able to output sparse probabilities. After deriving its properties, we show how its Jacobian can be efficiently computed, enabling its use in a network trained with backpropagation. Then, we propose a new smooth and convex loss function which is the sparsemax analogue of the logistic loss. We reveal an unexpected connection between this new loss and the Huber classification loss. We obtain promising empirical results in multi-label classification problems and in attention-based neural networks for natural language inference. For the latter, we achieve a similar performance as the traditional softmax, but with a selective, more compact, attention focus.

From Softmax to Sparsemax: A Sparse Model of Attention and Multi-Label Classification

TL;DR

This work introduces sparsemax, a differentiable activation that projects inputs onto the probability simplex to yield sparse output probabilities, enabling selective attention and multi-label predictions. It derives a closed-form threshold-based solution, the Jacobian for efficient backpropagation, and a convex sparsemax loss that mirrors logistic loss while promoting sparsity. The paper demonstrates competitive performance in multi-label classification and natural language inference tasks, with the sparsemax attention mechanism producing more compact and interpretable focus. Overall, sparsemax offers a principled alternative to softmax for scalable, sparse, and interpretable neural architectures with robust optimization properties.

Abstract

We propose sparsemax, a new activation function similar to the traditional softmax, but able to output sparse probabilities. After deriving its properties, we show how its Jacobian can be efficiently computed, enabling its use in a network trained with backpropagation. Then, we propose a new smooth and convex loss function which is the sparsemax analogue of the logistic loss. We reveal an unexpected connection between this new loss and the Huber classification loss. We obtain promising empirical results in multi-label classification problems and in attention-based neural networks for natural language inference. For the latter, we achieve a similar performance as the traditional softmax, but with a selective, more compact, attention focus.

Paper Structure

This paper contains 21 sections, 3 theorems, 37 equations, 4 figures, 4 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. The Sparsemax Transformation
  3. Definition
  4. Closed-Form Solution
  5. Basic Properties
  6. Two and Three-Dimensional Cases
  7. Jacobian of Sparsemax
  8. A Loss Function for Sparsemax
  9. Logistic Loss
  10. Sparsemax Loss
  11. Relation to the Huber Loss
  12. Generalization to Multi-Label Classification
  13. Experiments
  14. Label Proportion Estimation
  15. Multi-Label Classification on Benchmark Datasets
  16. ...and 6 more sections

Key Result

Proposition 1

The solution of Eq. eq:sparsemax is of the form: where $\tau : \mathbb{R}^K \rightarrow \mathbb{R}$ is the (unique) function that satisfies $\sum_j [z_j - \tau(\boldsymbol{z})]_+ = 1$ for every $\boldsymbol{z}$. Furthermore, $\tau$ can be expressed as follows. Let $z_{(1)} \ge z_{(2)} \ge \ldots \ge z_{(K)}$ be the sorted coordinates of $\boldsymb where $S(\boldsymbol{z}) := \{j \in [K]\,\,|\,\,

Figures (4)

  • Figure 1: Comparison of softmax and sparsemax in 2D (left) and 3D (two righmost plots).
  • Figure 2: Comparison between the sparsemax loss and other commonly used losses for binary classification.
  • Figure 3: Simulation results for the estimation of label posteriors, for uniform (top) and random mixtures (bottom). Shown are the mean squared error and the Jensen-Shannon divergence as a function of the document length, for the logistic and the sparsemax estimators.
  • Figure 4: Network diagram for the NL inference problem. The premise and hypothesis are both fed into (gated) RNNs. The NoAttention system replaces the attention part (in green) by a direct connection from the last premise state to the output (dashed violet line). The LogisticAttention, SoftAttention and SparseAttention systems have respectively independent logistics, a softmax, and a sparsemax-activated attention mechanism. In this example, $L=5$ and $N=9$.

Theorems & Definitions (6)

  • Proposition 1
  • proof
  • Proposition 2
  • proof
  • Proposition 3
  • proof