Modeling Deep Learning Based Privacy Attacks on Physical Mail

TL;DR

This work addresses the risk of hidden content in sealed envelopes being revealed by deep learning techniques. It introduces Neural-STE, a three-module network that explicitly models the image-formation process as a combination of perspective alignment, dehazing, and deblurring to recover the hidden content $J$ from the camera-captured image $I$, using the formulation $I(x)=L\;J\;\otimes\;h_x\,t(x)+A(x)$ and producing a refined estimate via $\hat{J}$. The authors provide a benchmark and show Neural-STE achieves superior recovery compared to baselines, while also enabling design of envelopes that counter such attacks by adjusting parameters like the blur kernel $h_x$, transmittance $t$, and surface reflectance $A$. The work highlights practical implications for mail privacy, offering a data-driven way to test envelope safety and informing material choices to mitigate privacy risks in real-world packaging.

Abstract

Mail privacy protection aims to prevent unauthorized access to hidden content within an envelope since normal paper envelopes are not as safe as we think. In this paper, for the first time, we show that with a well designed deep learning model, the hidden content may be largely recovered without opening the envelope. We start by modeling deep learning-based privacy attacks on physical mail content as learning the mapping from the camera-captured envelope front face image to the hidden content, then we explicitly model the mapping as a combination of perspective transformation, image dehazing and denoising using a deep convolutional neural network, named Neural-STE (See-Through-Envelope). We show experimentally that hidden content details, such as texture and image structure, can be clearly recovered. Finally, our formulation and model allow us to design envelopes that can counter deep learning-based privacy attacks on physical mail.

Figures (7)

• Figure 1: System setup and empirical image formation model. A color printed paper is put within the envelope (distance exaggerated for illustration). Deep learning-based privacy attacks aim to recover the hidden printed paper $J$ from the camera-captured envelope front surface image $I$. Our formulation models $I$ as a linear combination of the incident environment light $L$, blurred transmitted paper radiance $J$ and the envelope's front face reflected radiance $A$. We simplify inter-reflections and subsurface scattering and absorb them in $A$.
• Figure 2: Network architecture of our Neural-STE. It consists of three modules: WarpingNet $\mathcal{T}$, DehazingNet ($\mathcal{G}, \mathcal{F}_A, \mathcal{F}_t$) and RefineNet $\phi$. These modules together with the losses allow us to utilize our image formation model to effectively model privacy attacks on physical mail problem.
• Figure 3: Qualitative comparison. We show results from three different setups, with the easiest in red and the hardest in blue. We show two examples for each setup and the results of different methods are shown in the 2nd to 9th columns. The 1st column are the camera-captured envelope front face $I$. The 2nd to 4th columns are PSDNet guo2020learning, Pix2pix isola2017image and Pix2pixHD wang2018pix2pixHD, respectively. The 5th to 9th columns are degraded versions of the proposed Neural-STE, as described in Ablation study. Please see supplementary for larger versions of the images and more results.
• Figure 4: Intermediate results of Neural-STE. See supplementary for more images of the three setups.
• Figure 5: Visualization of the simulated camera-captured image when we tune the controllable parameters, i.e., the size of the blur kernel $h_x$, the surface reflected light $A$, the environment light $L$ and the envelope's transmittance $t$. Here we show an easy setup so that the hidden content is recognizable: as we tune each optical parameter from left to right, the hidden content becomes harder to recognize.