Resilience of Entropy Model in Distributed Neural Networks

TL;DR

This work investigates how entropy-coded distributed DNNs are affected by both intentional attacks and unintentional disturbances that alter bit-rate without necessarily compromising accuracy. It conducts a broad empirical study across multiple architectures and two entropy-prior models, showing that adversaries can inflate transmission overhead by up to ~95% and that high-frequency content largely drives the entropy-based compression. The authors introduce a standalone defense based on disentangling compression features in frequency and spatial domains and applying object-aware total variation denoising, which reduces overhead by about 9% on perturbed inputs with only ~2% accuracy loss, and remains robust under adaptive attacks. The approach is complementary to adversarial training and holds promise for enhancing reliability of entropy-coded distributed DNNs in edge-to-cloud systems, with code for reproducibility to follow.

Abstract

Distributed deep neural networks (DNNs) have emerged as a key technique to reduce communication overhead without sacrificing performance in edge computing systems. Recently, entropy coding has been introduced to further reduce the communication overhead. The key idea is to train the distributed DNN jointly with an entropy model, which is used as side information during inference time to adaptively encode latent representations into bit streams with variable length. To the best of our knowledge, the resilience of entropy models is yet to be investigated. As such, in this paper we formulate and investigate the resilience of entropy models to intentional interference (e.g., adversarial attacks) and unintentional interference (e.g., weather changes and motion blur). Through an extensive experimental campaign with 3 different DNN architectures, 2 entropy models and 4 rate-distortion trade-off factors, we demonstrate that the entropy attacks can increase the communication overhead by up to 95%. By separating compression features in frequency and spatial domain, we propose a new defense mechanism that can reduce the transmission overhead of the attacked input by about 9% compared to unperturbed data, with only about 2% accuracy loss. Importantly, the proposed defense mechanism is a standalone approach which can be applied in conjunction with approaches such as adversarial training to further improve robustness. Code will be shared for reproducibility.

Figures (6)

• Figure 1: The key concept of distributed . Left: Lightweight suffer from performance loss due to the limited computational resources; Middle: Edge computing often results in intolerable latency due to communication overhead; Right: Distributed deploy a small head model on the resource-limited device to achieve task-oriented compression and a large tail model on the server to decode compressed features and execute the rest of model.
• Figure 2: A general framework for entropy coding in neural data compression. It consists of an encoder-decoder structure to extract a compact latent representation $z$ and a prior model to estimate the distribution $P_Z(z)$. Left: During training, the prior model is jointly optimized with the encoder-decoder backbone. Right: During inference, the learned prior $P_Z(z)$ is used as side information for arithmetic encoding/decoding.
• Figure 3: Resilience of accuracy and data rate to 4 common corruptions.
• Figure 4: Resilience of accuracy and data rate to intentional interference
• Figure 5: Entropy models are sensitive to high-frequency features. From left to right: images from ImageNet validation set, bit rate maps and total variation maps.