LenslessFace: An End-to-End Optimized Lensless System for Privacy-Preserving Face Verification

TL;DR

The paper addresses privacy and efficiency challenges in face verification by moving from a reconstruct-then-verify paradigm to an end-to-end lensless approach that operates directly on encoded sensor captures. It jointly optimizes the lensless mask and the verification network, forming a differentiable pipeline S = g(I, M) that obviates intermediate reconstruction. Three core training strategies—face-center alignment in sensor space, an augmentation curriculum, and cross-modality distillation from RGB teachers—enable robust performance under translation, rotation, and background variations. Experiments in simulation and with a real lensless prototype show the method outperforms two-stage lensless verification and approaches RGB-based accuracy while preserving hardware-level privacy, though a gap between simulation and real-world deployment remains a limitation.

Abstract

Lensless cameras, innovatively replacing traditional lenses for ultra-thin, flat optics, encode light directly onto sensors, producing images that are not immediately recognizable. This compact, lightweight, and cost-effective imaging solution offers inherent privacy advantages, making it attractive for privacy-sensitive applications like face verification. Typical lensless face verification adopts a two-stage process of reconstruction followed by verification, incurring privacy risks from reconstructed faces and high computational costs. This paper presents an end-to-end optimization approach for privacy-preserving face verification directly on encoded lensless captures, ensuring that the entire software pipeline remains encoded with no visible faces as intermediate results. To achieve this, we propose several techniques to address unique challenges from the lensless setup which precludes traditional face detection and alignment. Specifically, we propose a face center alignment scheme, an augmentation curriculum to build robustness against variations, and a knowledge distillation method to smooth optimization and enhance performance. Evaluations under both simulation and real environment demonstrate our method outperforms two-stage lensless verification while enhancing privacy and efficiency. Project website: \url{lenslessface.github.io}.

Figures (12)

• Figure 1: Left: Comparison of different face verification approaches: lens-based (top) v.s. two-step lensless (middle) v.s. our optimized one-step lensless method. Our approach enhances privacy against software attacks as well as maintains robust face verification performance. Right: Challenges for robust lensless-based face verification as compared to RGB-based methods. Our robust lensless-based face verification is designed to accurately distinguish identities with encrypted captures, demonstrating resilience to variations in background, rotation, shift, and scales.
• Figure 1: Decoded Image quality for the varying number of plaintext attacks.
• Figure 2: Overview of our end-to-end optimization pipeline. The top pathway simulates lensless imaging using a learnable mask to encode a facial scene into a sensor capture. The capture after face-center alignment is fed into the lensless student model. In the bottom pathway, the same scene is aligned by preprocessing, and then processed by a trained RGB teacher model. The teacher's output features and ID labels supervise the training of both the lensless student model and the learnable mask.
• Figure 3: Center alignment for lensless capture of facial scenes.
• Figure 4: Curriculum learning for augmentation. The augmentation intensity gradually increases with training epochs.