Encrypted Large Model Inference: The Equivariant Encryption Paradigm
James Buban, Hongyang Zhang, Claudio Angione, Harry Yang, Ahmad Farhan, Seyfal Sultanov, Michael Du, Xuran Ma, Zihao Wang, Yue Zhao, Arria Owlia, Fielding Johnston, Patrick Colangelo
TL;DR
The paper tackles privacy concerns in distributed large-scale model inference by proposing Equivariant Encryption (EE), a targeted, layer-wise encryption paradigm that preserves the exact functionality of linear operations and a subset of nonlinear activations while enabling encrypted input, intermediate states, and outputs to be processed with near-plaintext efficiency. EE is positioned as a practical middle ground between differential privacy, SMPC, and homomorphic encryption, offering strong confidentiality with negligible runtime overhead by transforming only critical internal representations rather than the entire computation graph. The work provides a formal definition of EE, situates it within DP/SMPC/HE ecosystems, analyzes threat models and potential inversion attacks, and benchmarks fidelity and latency against standard inference pipelines in decentralized settings. Overall, EE aims to deliver “always-encrypted” inference suitable for LLM serving, vision, and RAG workflows, bridging data confidentiality with the performance demands of modern, large-scale inference in distributed environments. Specifically, EE maintains recoverability $p=\,\mathrm{decrypt}(\mathrm{encrypt}(p))$ and equivariance $\mathrm{decrypt}(F(\mathrm{encrypt}(p)))=F(p)$ for linear operations $F$ and a chosen set of nonlinear activations, enabling secure, scalable, and efficient blind inference.
Abstract
Large scale deep learning model, such as modern language models and diffusion architectures, have revolutionized applications ranging from natural language processing to computer vision. However, their deployment in distributed or decentralized environments raises significant privacy concerns, as sensitive data may be exposed during inference. Traditional techniques like secure multi-party computation, homomorphic encryption, and differential privacy offer partial remedies but often incur substantial computational overhead, latency penalties, or limited compatibility with non-linear network operations. In this work, we introduce Equivariant Encryption (EE), a novel paradigm designed to enable secure, "blind" inference on encrypted data with near zero performance overhead. Unlike fully homomorphic approaches that encrypt the entire computational graph, EE selectively obfuscates critical internal representations within neural network layers while preserving the exact functionality of both linear and a prescribed set of non-linear operations. This targeted encryption ensures that raw inputs, intermediate activations, and outputs remain confidential, even when processed on untrusted infrastructure. We detail the theoretical foundations of EE, compare its performance and integration complexity against conventional privacy preserving techniques, and demonstrate its applicability across a range of architectures, from convolutional networks to large language models. Furthermore, our work provides a comprehensive threat analysis, outlining potential attack vectors and baseline strategies, and benchmarks EE against standard inference pipelines in decentralized settings. The results confirm that EE maintains high fidelity and throughput, effectively bridging the gap between robust data confidentiality and the stringent efficiency requirements of modern, large scale model inference.