Permutation Randomization on Nonsmooth Nonconvex Optimization: A Theoretical and Experimental Study

TL;DR

This work investigates the role of permutation randomization in gradient-based optimization for nonsmooth nonconvex problems, offering both theoretical and empirical insights. The authors prove that permutation randomization disrupts contraction, enabling the iterates to collectively cover a closed cube surrounding the global optimum while preserving the original optimizer's convergence rate. They demonstrate, under sufficiently many iterations, that randomized strategies can overcome the limitations of deterministic methods and approximate the global optimum more reliably. Empirically, randomized ADAM outperforms several baselines on deep nonlinear matrix factorization, deep belief networks, and noisy optimization tasks, with strong reconstruction accuracy and reliable consistency metrics, supporting the theoretical claims. Overall, the work lays a foundation for extending analytics to a broader class of randomized optimization techniques.

Abstract

While gradient-based optimizers that incorporate randomization often showcase superior performance on complex optimization, the theoretical foundations underlying this superiority remain insufficiently understood. A particularly pressing question has emerged: What is the role of randomization in dimension-free nonsmooth nonconvex optimization? To address this gap, we investigate the theoretical and empirical impact of permutation randomization within gradient-based optimization frameworks, using it as a representative case to explore broader implications. From a theoretical perspective, our analyses reveal that permutation randomization disrupts the shrinkage behavior of gradient-based optimizers, facilitating continuous convergence toward the global optimum given a sufficiently large number of iterations. Additionally, we prove that permutation randomization can preserve the convergence rate of the underlying optimizer. On the empirical side, we conduct extensive numerical experiments comparing permutation-randomized optimizer against three baseline methods. These experiments span tasks such as training deep neural networks with stacked architectures and optimizing noisy objective functions. The results not only corroborate our theoretical insights but also highlight the practical benefits of permutation randomization. In summary, this work delivers both rigorous theoretical justification and compelling empirical evidence for the effectiveness of permutation randomization. Our findings and evidence lay a foundation for extending analytics to encompass a wide array of randomization.

Figures (8)

• Figure 1: An illustration of a gradient-based optimizer incorporating permutation randomization can cover a closed cube of global optimum when the maximum iteration $T$ is sufficiently large.
• Figure 2: The reconstruction loss comparison of randomized ADAM and the other three peer optimizers within two thousand iterations of randomly selected two subjects at the first layer.
• Figure 3: The reconstruction loss comparison of randomized ADAM and the other three peer optimizers within two thousand iterations of randomly selected two subjects at the second layer.
• Figure 4: The averaged reconstruction loss comparison of randomized ADAM and the other three peer optimizers within two thousand iterations across all subjects at first and second layers, respectively.
• Figure 5: An illustration of time-consumption and consistency comparisons of randomized ADAM and other peer optimizers. The box plots in (a) and (b) represent the time consumption of four algorithms using all subjects; in Figure \ref{['fig:fig5']}(c) provides the ICC values to demonstrate an overall consistency.

Theorems & Definitions (15)

• Definition 2.1
• Definition 2.2
• Definition 2.3
• Definition 2.4
• Definition 2.5
• Definition 2.6
• Definition 2.7
• Lemma 3.1
• Theorem 3.2
• Lemma 4.1