Deep Multimodal Subspace Clustering Networks

TL;DR

This paper tackles unsupervised clustering of multimodal data by introducing a CNN-based framework that embeds all modalities into a latent space while enforcing self-expressiveness via a trainable layer $Θ_s$, producing an affinity matrix for clustering. It presents two fusion paradigms: spatial fusion (early/intermediate/late) and affinity fusion (shared self-expressive layer across modalities), and trains end-to-end with a loss that combines self-expression and reconstruction terms. The approach achieves state-of-the-art results on multiple multimodal datasets, notably achieving near-perfect metrics on Extended Yale-B with affinity fusion ($ACC\approx99\%, NMI\approx98.89\%, ARI\approx98.38\%$). These findings demonstrate the effectiveness of joint multimodal representation learning and shared affinity learning for unsupervised subspace clustering across diverse domains, with practical implications for multimodal biometric and vision tasks.

Abstract

We present convolutional neural network (CNN) based approaches for unsupervised multimodal subspace clustering. The proposed framework consists of three main stages - multimodal encoder, self-expressive layer, and multimodal decoder. The encoder takes multimodal data as input and fuses them to a latent space representation. The self-expressive layer is responsible for enforcing the self-expressiveness property and acquiring an affinity matrix corresponding to the data points. The decoder reconstructs the original input data. The network uses the distance between the decoder's reconstruction and the original input in its training. We investigate early, late and intermediate fusion techniques and propose three different encoders corresponding to them for spatial fusion. The self-expressive layers and multimodal decoders are essentially the same for different spatial fusion-based approaches. In addition to various spatial fusion-based methods, an affinity fusion-based network is also proposed in which the self-expressive layer corresponding to different modalities is enforced to be the same. Extensive experiments on three datasets show that the proposed methods significantly outperform the state-of-the-art multimodal subspace clustering methods.

Figures (10)

• Figure 1: An overview of the proposed deep multimodal subspace clustering framework. Note that the network consists of three blocks: a multimodal encoder, a self-expressive layer, and a multimodal decoder. The weights in the self-expressive layer, $\boldsymbol{\Theta}_s$, are used to construct the affinity matrix. We present several models for the multimodal encoder.
• Figure 2: An overview of the DSC framework proposed in deepsc17nips for unimodal subspace clustering.
• Figure 3: Different network architectures corresponding to (a) early fusion, (b) intermediate fusion, and (c) late fusion. Note that in all the spatial fusion-based networks (a)-(c), the overall structure for the self-expressive layer and the multimodal decoder remain the same. (d) Network architecture corresponding to affinity fusion. In this case, the encoder and decoder are trained separately for each modality, but are forced to have the same self-expressive layer.
• Figure 4: In spatial fusion methods each location of the fusion is related to the input values at the same location. In this especial case, the facial components (i.e. eyes, nose and mouth) are aligned across all the modalities (i.e. DP, S0, S1, S2, Visible).
• Figure 5: An example of affinity fusion. Affinities corresponding to different modalities are combined to have only a single shared affinity. This method does not relay on spatial relation across different modalities. Instead, it aggregates the similarities among data points across different modalities and returns a shared affinity.