A Sensitivity Analysis of (and Practitioners' Guide to) Convolutional Neural Networks for Sentence Classification

TL;DR

The paper investigates how sensitive a simple one-layer CNN is to architectural choices in sentence classification, aiming to identify which design decisions matter most in practice. Through extensive replication and evaluation on nine datasets, it reveals that word-vector choices, filter region size, and the number of feature maps significantly influence performance, while pooling and regularization tend to be more robust or less impactful. It offers concrete, empirically grounded guidance for practitioners, including line-search strategies, preferred pooling, and reasonable ranges for key hyperparameters. The work positions a simple one-layer CNN as a strong, reliable baseline for real-world sentence classification tasks and highlights the importance of variance-aware evaluation in hyperparameter tuning.

Abstract

Convolutional Neural Networks (CNNs) have recently achieved remarkably strong performance on the practically important task of sentence classification (kim 2014, kalchbrenner 2014, johnson 2014). However, these models require practitioners to specify an exact model architecture and set accompanying hyperparameters, including the filter region size, regularization parameters, and so on. It is currently unknown how sensitive model performance is to changes in these configurations for the task of sentence classification. We thus conduct a sensitivity analysis of one-layer CNNs to explore the effect of architecture components on model performance; our aim is to distinguish between important and comparatively inconsequential design decisions for sentence classification. We focus on one-layer CNNs (to the exclusion of more complex models) due to their comparative simplicity and strong empirical performance, which makes it a modern standard baseline method akin to Support Vector Machine (SVMs) and logistic regression. We derive practical advice from our extensive empirical results for those interested in getting the most out of CNNs for sentence classification in real world settings.

Figures (8)

• Figure 1: Illustration of a CNN architecture for sentence classification. We depict three filter region sizes: 2, 3 and 4, each of which has 2 filters. Filters perform convolutions on the sentence matrix and generate (variable-length) feature maps; 1-max pooling is performed over each map, i.e., the largest number from each feature map is recorded. Thus a univariate feature vector is generated from all six maps, and these 6 features are concatenated to form a feature vector for the penultimate layer. The final softmax layer then receives this feature vector as input and uses it to classify the sentence; here we assume binary classification and hence depict two possible output states.
• Figure 2: Density curve of accuracy using static and non-static word2vec-CNN
• Figure 3: Effect of the region size (using only one).
• Figure 4: Effect of the number of feature maps.
• Figure 5: Effect of dropout rate. The accuracy when the dropout rate is 0.9 on the Opi dataset is about 10% worse than baseline, and thus is not visible on the figure at this point.