Do We Always Need the Simplicity Bias? Looking for Optimal Inductive Biases in the Wild

TL;DR

This work questions the universality of the $ReLU$-induced simplicity bias in neural networks and introduces a meta-learning framework to discover dataset-specific activation functions, thereby tailoring inductive biases to tasks. By parametrizing activations with splines and optimizing them bi-linearly (inner model training and outer activation learning) across episodic tasks, the authors quantify model complexity via total variation and demonstrate that higher-complexity activations can improve generalization on tabular data, regression, shortcut learning, and algorithmic grokking tasks, while image classifications remain well served by ReLU-like biases. The study reports that learned activations often mimic known alternatives (e.g., GeLUs) on images but uncover novel shapes in other domains, indicating that simplicity bias is not universally optimal. Importantly, learned activations transfer across related tasks, suggesting practical avenues for designing domain-tailored inductive biases beyond naive scaling. Overall, the paper highlights that a richer landscape of inductive biases, accessed through learnable activations, can improve data efficiency and generalization in diverse settings, while calling for systematic characterization of biases beyond complexity.

Abstract

Neural architectures tend to fit their data with relatively simple functions. This "simplicity bias" is widely regarded as key to their success. This paper explores the limits of this principle. Building on recent findings that the simplicity bias stems from ReLU activations [96], we introduce a method to meta-learn new activation functions and inductive biases better suited to specific tasks. Findings: We identify multiple tasks where the simplicity bias is inadequate and ReLUs suboptimal. In these cases, we learn new activation functions that perform better by inducing a prior of higher complexity. Interestingly, these cases correspond to domains where neural networks have historically struggled: tabular data, regression tasks, cases of shortcut learning, and algorithmic grokking tasks. In comparison, the simplicity bias induced by ReLUs proves adequate on image tasks where the best learned activations are nearly identical to ReLUs and GeLUs. Implications: Contrary to popular belief, the simplicity bias of ReLU networks is not universally useful. It is near-optimal for image classification, but other inductive biases are sometimes preferable. We showed that activation functions can control these inductive biases, but future tailored architectures might provide further benefits. Advances are still needed to characterize a model's inductive biases beyond "complexity", and their adequacy with the data.

Figures (31)

• Figure 1: (1) We modulate the inductive bias of neural architectures by learning novel activation functions that improve generalization on specific datasets. (2) With this tool, we study the relation between model accuracy and complexity. We identify tasks where the simplicity bias of ReLU architectures is suboptimal.
• Figure 2: Test accuracy on image datasets. (a) For classification tasks, all models perform similarly, suggesting that the inductive biases of ReLUs are well suited to these datasets. (b) For regression tasks, models with learned activations perform better, especially from an initialization as zeros, which enables the discovery of completely novel activation functions.
• Figure 3: Activation functions learned for mnist. For a classification task, the activation learned from a ReLU resembles the popular GeLUs. For a regression task, the learned activations contain irregularities that help a network represent complex functions. See \ref{['fig:imgActAll']} for similar results on other datasets.
• Figure 4: Accuracy vs. complexity on image datasets. Each marker is a model with different hyperparameters and ReLUs ([0.75]●) or learned activations initialized as ReLUs ([0.75]■) or as zeros ([0.75]■). For classification (a), ReLUs are close to best. Activations optimized from ReLUs only improve the accuracy slightly, corresponding to the GeLU-like function in \ref{['fig:imgAct']}. For regression (b), new activations (learned from zeros) are best. Moreover, accuracy and complexity are clearly correlated only for regression. This supports the hypothesis that regression is more complex than classification and thus benefits from alternatives to the ReLUs' simplicity bias. See \ref{['fig:imgPlotsAll']} for similar results on other datasets.
• Figure 5: Comparison of model types over 16 tabular datasets. Vanilla MLPs often perform worse than decision trees, but adjusting their inductive biases with learned activation functions (AFs) eliminates this gap. The input activation functions (IAFs) enable even better performance. See \ref{['fig:tabAllDatasets']} for results per dataset.