On Polynomial Approximations for Privacy-Preserving and Verifiable ReLU Networks
Ramy E. Ali, Jinhyun So, A. Salman Avestimehr
TL;DR
This work tackles privacy-preserving and verifiable inference for ReLU networks outsourced to untrusted clouds by replacing non-polynomial activations with polynomials. It analyzes the feasibility of approximating ReLU with degree-2 polynomials under integer-coefficient or finite-field constraints, showing that the classic ReLU cannot be uniformly approximated on $I=[-1,1]$ with integer coefficients, but a scaled ReLU with even scale can, motivating the surrogate $\sigma_{\mathrm{poly}}(x)=x^2+ax$. Through minimax and uniform-approximation theory, it derives $\sigma_{\mathrm{poly}}$ and demonstrates, across CIFAR-10/100 and Tiny-ImageNet with architectures like LiU’s CNN, NIN, LeNet, and VGG-16, that the polynomial activation yields substantial accuracy gains over the square function and close-to-ReLU performance, up to 10.4% in some settings. This establishes a practically effective pathway for secure, private, and verifiable inference using finite-field arithmetic in DNNs and points to future work on broader architectures and coefficient choices.
Abstract
Outsourcing deep neural networks (DNNs) inference tasks to an untrusted cloud raises data privacy and integrity concerns. While there are many techniques to ensure privacy and integrity for polynomial-based computations, DNNs involve non-polynomial computations. To address these challenges, several privacy-preserving and verifiable inference techniques have been proposed based on replacing the non-polynomial activation functions such as the rectified linear unit (ReLU) function with polynomial activation functions. Such techniques usually require polynomials with integer coefficients or polynomials over finite fields. Motivated by such requirements, several works proposed replacing the ReLU function with the square function. In this work, we empirically show that the square function is not the best degree-2 polynomial that can replace the ReLU function even when restricting the polynomials to have integer coefficients. We instead propose a degree-2 polynomial activation function with a first order term and empirically show that it can lead to much better models. Our experiments on the CIFAR and Tiny ImageNet datasets on various architectures such as VGG-16 show that our proposed function improves the test accuracy by up to 10.4% compared to the square function.