k-Sparse Autoencoders
Alireza Makhzani, Brendan Frey
TL;DR
The paper tackles the challenge of obtaining sparse representations without complex penalties or nonlinearities. It introduces the k-sparse autoencoder, a linear autoencoder that preserves only the top-k hidden activations, enabling fast training and effective sparse coding. Through analysis linking it to sparse coding with incoherent dictionaries and ITI-like recovery, the work provides theoretical and empirical justification for the approach. Empirically, k-sparse autoencoders yield strong unsupervised and supervised performance on MNIST and NORB, and serve as an effective pretraining mechanism for both shallow and deep networks.
Abstract
Recently, it has been observed that when representations are learnt in a way that encourages sparsity, improved performance is obtained on classification tasks. These methods involve combinations of activation functions, sampling steps and different kinds of penalties. To investigate the effectiveness of sparsity by itself, we propose the k-sparse autoencoder, which is an autoencoder with linear activation function, where in hidden layers only the k highest activities are kept. When applied to the MNIST and NORB datasets, we find that this method achieves better classification results than denoising autoencoders, networks trained with dropout, and RBMs. k-sparse autoencoders are simple to train and the encoding stage is very fast, making them well-suited to large problem sizes, where conventional sparse coding algorithms cannot be applied.