No Training Required: Exploring Random Encoders for Sentence Classification

TL;DR

The paper investigates whether randomly parameterized encoders can yield competitive sentence representations built solely from pre-trained word embeddings, without training the encoder. It introduces three random architectures (BOREP, Random LSTMs, ESNs) and evaluates them with a fixed logistic regression classifier on SentEval tasks, comparing against InferSent and SkipThought. Findings show random encoders often beat simple baselines, with ESNs performing best among the random methods, and that training gains over random baselines are modest on the standard benchmarks. The work emphasizes robust baselines, the impact of dimensionality, and the importance of pre-trained word embeddings as the primary source of signal, urging the community to focus on tasks that require more than what word embeddings alone provide.

Abstract

We explore various methods for computing sentence representations from pre-trained word embeddings without any training, i.e., using nothing but random parameterizations. Our aim is to put sentence embeddings on more solid footing by 1) looking at how much modern sentence embeddings gain over random methods---as it turns out, surprisingly little; and by 2) providing the field with more appropriate baselines going forward---which are, as it turns out, quite strong. We also make important observations about proper experimental protocol for sentence classification evaluation, together with recommendations for future research.

Figures (1)

• Figure 1: Performance while varying dimensionality, for the three random sentence encoders over all ten downstream tasks.