Real-Time Privacy Preservation for Robot Visual Perception

TL;DR

This work tackles real-time privacy in robotic vision by introducing Privacy-Constrained Video Streaming (PCVS), which enforces user-defined privacy specifications $\Phi$ through per-frame detection by a vision-language model, conformal calibration, and selective concealment. A video abstraction in the form of a labeled Markov chain supports efficient, real-time probabilistic guarantees $\mathcal{PG}_k(\mathcal{A}_k \models \Phi)$ for frame sequences, updated incrementally as new frames arrive. Conformal prediction provides statistically valid lower bounds on detection reliability, while the abstraction enables scalable verification for long video streams. The framework is demonstrated on indoor, ground, and aerial robotic platforms, showing high privacy-preservation rates ($\mathcal{PG}$ typically $0.80$–$0.97$) with real-time performance (up to ~14 fps on GPU) and preserving non-private visual information. Overall, PCVS offers a principled, real-time solution to privacy in robot perception with formal guarantees and practical applicability.

Abstract

Many robots (e.g., iRobot's Roomba) operate based on visual observations from live video streams, and such observations may inadvertently include privacy-sensitive objects, such as personal identifiers. Existing approaches for preserving privacy rely on deep learning models, differential privacy, or cryptography. They lack guarantees for the complete concealment of all sensitive objects. Guaranteeing concealment requires post-processing techniques and thus is inadequate for real-time video streams. We develop a method for privacy-constrained video streaming, PCVS, that conceals sensitive objects within real-time video streams. PCVS takes a logical specification constraining the existence of privacy-sensitive objects, e.g., never show faces when a person exists. It uses a detection model to evaluate the existence of these objects in each incoming frame. Then, it blurs out a subset of objects such that the existence of the remaining objects satisfies the specification. We then propose a conformal prediction approach to (i) establish a theoretical lower bound on the probability of the existence of these objects in a sequence of frames satisfying the specification and (ii) update the bound with the arrival of each subsequent frame. Quantitative evaluations show that PCVS achieves over 95 percent specification satisfaction rate in multiple datasets, significantly outperforming other methods. The satisfaction rate is consistently above the theoretical bounds across all datasets, indicating that the established bounds hold. Additionally, we deploy PCVS on robots in real-time operation and show that the robots operate normally without being compromised when PCVS conceals objects.

Figures (11)

• Figure 1: Pipeline of Privacy-Constrained Video Streaming:(Step 1) Given a privacy specification $\Phi$, we define a set $AP$ of atomic propositions describing privacy-sensitive objects. (Step 2) Given an incoming frame $\mathcal{F}_k$ from the video, the method uses a vision-language model (VLM) to detect sensitive objects in the frame. Each detection is associated with a confidence score from the VLM. The method calibrates a confidence score to a per-frame probability bound for correct detection via a calibration function $f_C$, as in Equation \ref{['eq: calibration']}. (Step 3) The method builds an abstract model $\mathcal{A}_k$ representing object detections and their probability bounds in the frame sequence $\mathcal{F}_1,...,\mathcal{F}_k$ via Algorithm \ref{['algo: rt-abstract']}. Then, it computes a theoretical bound for the probability of $\mathcal{A}_k$ satisfying $\Phi$, i.e., a probabilistic guarantee $\mathcal{PG}_k(\mathcal{A}_k \models \Phi)$ using Equation \ref{['eq: sequence-guarantee']}. (Step 4) If $\mathcal{PG}_k(\mathcal{A}_k \models \Phi)$ is below a user-given privacy threshold $\lambda$, the method removes a subset of sensitive objects and goes back to Step 2 to recompute a guarantee. (Step 5) If $\mathcal{PG}_k(\mathcal{A}_k \models \Phi)$ is above $\lambda$, the method adds $\mathcal{F}_k$ back to the stream and proceeds to Step 1 with the next incoming frame. We number each step in blue.
• Figure 2: A running example on how to compute the probabilistic guarantee via video abstraction.
• Figure 3: We present the demonstrations for indoor robot navigation (top left), ground robot navigation (top right), and urban drone monitoring (bottom left). The indoor robot and the ground robot are shown in the bottom right. Scenes with an 'x' in a red circle contain privacy-sensitive objects and our method successfully conceals them. All demonstrations effectively maintain privacy above the user-given privacy threshold of 0.80, denoted as $\mathcal{PG}_k(\Phi) > 0.80$.
• Figure 4: A sample control policy for the ground robot. Each transition is associated with an (input, output) tuple.
• Figure 5: PCVS effectively maintains privacy in long-horizon videos and complex privacy specifications. In Figure \ref{['fig:eval_privacy_preservation_by_length']}, PCVS consistently preserves privacy, achieving an average Privacy Preservation Success Ratio of $0.97$ across various video lengths. In Figure \ref{['fig:eval_privacy_preservation_by_complexity']}, we show that PCVS consistently upholds privacy regardless of the complexity of specifications with an average Privacy Preservation Success Ratio of $0.94$.

Theorems & Definitions (8)

• Definition 1: Probabilistic Guarantee on a Frame Sequence
• Theorem 1
• Definition 2: Video Abstraction
• Definition 3: Safety Property
• Theorem 2
• proof
• Proposition 1
• proof