Targeted Therapy in Data Removal: Object Unlearning Based on Scene Graphs
Chenhan Zhang, Benjamin Zi Hao Zhao, Hassan Asghar, Dali Kaafar
TL;DR
This work tackles granular privacy by introducing a scene-graph–based object unlearning framework that can selectively remove specific objects from images generated by SG2I models while preserving remaining content. It formalizes object unlearning, leveraging scene graphs to map user requests to actionable model edits and proposes three unlearning methods (negative guidance, masking, and influence-function redaction) to avoid full retraining. Extensive experiments on Visual Genome across image reconstruction and synthesis demonstrate that object-masked fine-tuning offers the best trade-off between unlearning effectiveness and fidelity, with strong robustness to indirect leakage attacks. The approach significantly speeds up unlearning compared to retraining and provides a scalable path for privacy-compliant MLaaS without sacrificing overall data utility.
Abstract
Users may inadvertently upload personally identifiable information (PII) to Machine Learning as a Service (MLaaS) providers. When users no longer want their PII on these services, regulations like GDPR and COPPA mandate a right to forget for these users. As such, these services seek efficient methods to remove the influence of specific data points. Thus the introduction of machine unlearning. Traditionally, unlearning is performed with the removal of entire data samples (sample unlearning) or whole features across the dataset (feature unlearning). However, these approaches are not equipped to handle the more granular and challenging task of unlearning specific objects within a sample. To address this gap, we propose a scene graph-based object unlearning framework. This framework utilizes scene graphs, rich in semantic representation, transparently translate unlearning requests into actionable steps. The result, is the preservation of the overall semantic integrity of the generated image, bar the unlearned object. Further, we manage high computational overheads with influence functions to approximate the unlearning process. For validation, we evaluate the unlearned object's fidelity in outputs under the tasks of image reconstruction and image synthesis. Our proposed framework demonstrates improved object unlearning outcomes, with the preservation of unrequested samples in contrast to sample and feature learning methods. This work addresses critical privacy issues by increasing the granularity of targeted machine unlearning through forgetting specific object-level details without sacrificing the utility of the whole data sample or dataset feature.