Accurate Low-Degree Polynomial Approximation of Non-polynomial Operators for Fast Private Inference in Homomorphic Encryption

TL;DR

The paper tackles the latency-accuracy gap in Fully Homomorphic Encryption (FHE) for ML by replacing non-polynomial operators with low-degree Polynomial Approximated Functions (PAFs). It introduces SMART-PAF, a framework incorporating four techniques—Coefficient Tuning, Progressive Approximation, Alternate Training, and Dynamic/Static Scaling—to enable 8–14-degree PAFs while preserving or improving model accuracy. The approach yields strong Pareto-frontier gains, achieving up to 7.81× speedups with comparable or higher accuracy than higher-degree PAF baselines on ResNet-18/ImageNet-1k, and demonstrates substantial improvements across VGG-19/CiFar-10 as well. This work offers a practical pathway for accurate, privacy-preserving inference compatible with CKKS-based FHE deployments, supported by public code.

Abstract

As machine learning (ML) permeates fields like healthcare, facial recognition, and blockchain, the need to protect sensitive data intensifies. Fully Homomorphic Encryption (FHE) allows inference on encrypted data, preserving the privacy of both data and the ML model. However, it slows down non-secure inference by up to five magnitudes, with a root cause of replacing non-polynomial operators (ReLU and MaxPooling) with high-degree Polynomial Approximated Function (PAF). We propose SmartPAF, a framework to replace non-polynomial operators with low-degree PAF and then recover the accuracy of PAF-approximated model through four techniques: (1) Coefficient Tuning (CT) -- adjust PAF coefficients based on the input distributions before training, (2) Progressive Approximation (PA) -- progressively replace one non-polynomial operator at a time followed by a fine-tuning, (3) Alternate Training (AT) -- alternate the training between PAFs and other linear operators in the decoupled manner, and (4) Dynamic Scale (DS) / Static Scale (SS) -- dynamically scale PAF input value within (-1, 1) in training, and fix the scale as the running max value in FHE deployment. The synergistic effect of CT, PA, AT, and DS/SS enables SmartPAF to enhance the accuracy of the various models approximated by PAFs with various low degrees under multiple datasets. For ResNet-18 under ImageNet-1k, the Pareto-frontier spotted by SmartPAF in latency-accuracy tradeoff space achieves 1.42x ~ 13.64x accuracy improvement and 6.79x ~ 14.9x speedup than prior works. Further, SmartPAF enables a 14-degree PAF (f1^2 g_1^2) to achieve 7.81x speedup compared to the 27-degree PAF obtained by minimax approximation with the same 69.4% post-replacement accuracy. Our code is available at https://github.com/EfficientFHE/SmartPAF.

Figures (10)

• Figure 1: SMART-PAF replaces ReLU and MaxPooling by low-degree PAFs, and achieves better accuracy-latency Pareto-frontier than prior works Lee2021PreciseAOMinimax_approximationcheon2020efficient on ResNet-18 (ImageNet-1k).
• Figure 2: Overview of the FHE-base ML inference where original non-polynomial operators are replaced by Polynomial Approximated Activation (PAF).
• Figure 3: Coefficient Tuning (CT) uses profiled distribution to tune PAF coefficients to generate more accurate results on a reduced smaller input range (high-probability range in profiled data distribution), leading to closer-to-optimal initialization, higher accuracy, and lower training time
• Figure 4: Progressive Approximation (PA) progressively replaces non-polynomial operators, one layer at a time followed by coefficients fine-tuning, to guarantee an SGD-optimizable convex regression problem shown in Eq. \ref{['equ:regression']}, enabling training convergence for replacing the targeted model with PAFs of arbitrary degrees.
• Figure 5: Alternate Training (AT) fine-tunes PAF coefficients and parameters of other layers separately in an alternate manner.