Analyzing Monotonic Linear Interpolation in Neural Network Loss Landscapes

TL;DR

This paper challenges the universality of the Monotonic Linear Interpolation (MLI) property by showing that, while SGD often yields monotonic loss decay along the line from initialization to a converged parameter, this is not guaranteed across modern architectures, optimizers, or training tricks. It develops a theoretical framework linking the geometry of the interpolation path in function space (via Gauss length) and the weight-space trajectory to the persistence of MLI under mean squared error, proving a sufficient condition for MLI in wide or two-layer linear regimes. The work systematically demonstrates that large weight movements, batch normalization, and adaptive optimization can produce non-monotonic interpolations, revealing a global loss-landscape structure where barriers and curvature interact with optimization dynamics. The findings have implications for understanding optimization ease, generalization, and the geometry of neural network loss landscapes beyond the classical lazy-training or near-linear regimes. Overall, the paper enriches our conceptual toolkit for analyzing loss landscapes and highlights directions for future work on the global properties of neural network optimization.

Abstract

Linear interpolation between initial neural network parameters and converged parameters after training with stochastic gradient descent (SGD) typically leads to a monotonic decrease in the training objective. This Monotonic Linear Interpolation (MLI) property, first observed by Goodfellow et al. (2014) persists in spite of the non-convex objectives and highly non-linear training dynamics of neural networks. Extending this work, we evaluate several hypotheses for this property that, to our knowledge, have not yet been explored. Using tools from differential geometry, we draw connections between the interpolated paths in function space and the monotonicity of the network - providing sufficient conditions for the MLI property under mean squared error. While the MLI property holds under various settings (e.g. network architectures and learning problems), we show in practice that networks violating the MLI property can be produced systematically, by encouraging the weights to move far from initialization. The MLI property raises important questions about the loss landscape geometry of neural networks and highlights the need to further study their global properties.

Figures (29)

• Figure 1: Monotonic linear interpolation for a ResNet-20 trained on CIFAR-10 from initialization to an optimum (red) and from an unrelated initialization to the same optimum (blue). On the left, we show a 2D slice of the loss landscape, defined by the two initializations and optimum, along with the optimization trajectory projected onto the plane (orange). On the right, we show the interpolated loss curves, with training loss shown relative to the proportion of distance travelled to the optimum.
• Figure 2: Training loss over the linear interpolation connecting initial and final parameters. Each curve represents a network trained on CIFAR-10 with different hyperparameter configurations (achieving at least 1.0 training loss). The MLI property holds for networks trained with SGD, but often fails for networks trained with Adam.
• Figure 3: 2D projections (computed with PCA) of logit interpolations for fully-connected networks trained on Fashion-MNIST. Both networks achieve near-perfect final training accuracy. However, the first one (left) interpolates monotonically while the second one (right) does not. The only difference between these two networks is that the second was trained using batch normalization while the first was not.
• Figure 4: Training loss over linear interpolation of deep autoencoders trained on MNIST using SGD and Adam. Each interpolation line is for a training configuration with different hyperparameters (achieving better than 30 training loss).
• Figure 5: For each MNIST & Fashion-MNIST classifier, we compute the minimum $\Delta$ such that the interpolated loss is $\Delta$-monotonic. We plot models trained with a learning rate of 0.1 and 0.0001 in the top and bottom rows respectively. On the left, we compare the distance moved in the weight space. On the right, we compare the Gauss length of the interpolated network outputs. Blue points represent networks where the MLI property holds and orange points are networks where the MLI property fails.

Theorems & Definitions (14)

• definition 1
• definition 2: Gauss length
• theorem 1: Small Gauss length gives monotonicity
• definition 2: Gauss length
• theorem 1: Small Gauss length gives monotonicity
• lemma 1
• proof
• proof
• lemma 2
• lemma 3