Smooth Approximations of the Rounding Function
Stanislav Semenov
TL;DR
Classical rounding is non-differentiable, hindering gradient-based optimization. The paper introduces two fully differentiable smooth rounding constructions: (i) a sigmoid-difference based rounding with localized windows and (ii) a density-based, sigmoid-derivative weighted average that interpolates between nearby integers. Both converge pointwise to $\mathrm{round}(x)$ as the sharpness parameter $k$ grows, with explicit gradient formulas provided. Computational efficiency is achieved by truncating sums to a small neighborhood (typically $M\approx5$ neighbors), making the methods practical for large-scale differentiable pipelines and differentiable optimization tasks.
Abstract
We propose novel smooth approximations to the classical rounding function, suitable for differentiable optimization and machine learning applications. Our constructions are based on two approaches: (1) localized sigmoid window functions centered at each integer, and (2) normalized weighted sums of sigmoid derivatives representing local densities. The first method approximates the step-like behavior of rounding through differences of shifted sigmoids, while the second method achieves smooth interpolation between integers via density-based weighting. Both methods converge pointwise to the classical rounding function as the sharpness parameter k tends to infinity, and allow controlled trade-offs between smoothness and approximation accuracy. We demonstrate that by restricting the summation to a small set of nearest integers, the computational cost remains low without sacrificing precision. These constructions provide fully differentiable alternatives to hard rounding, which are valuable in contexts where gradient-based methods are essential.