Responsible Visual Editing

TL;DR

The paper defines responsible visual editing as minimally altering an image to remove or mitigate risky concepts across safety, fairness, and privacy. It introduces CoEditor, a two-stage cognitive framework that uses PCP to locate modification targets and BCP to plan edits, leveraging a large multimodal model (GPT-4v) and Semantic-SAM for guidance, and applies Stable Diffusion Inpainting for final edits. To support evaluation, it presents AltBear, a safe teddy-bear–based dataset that mirrors real harmful content, along with machine and human metrics for success and visual similarity, and markers to prevent misuse. Experimental results show CoEditor substantially surpasses baselines (InstructPix2pix and InstructDiffusion) in both quantitative and qualitative assessments, including general editing, with strong consistency between AltBear and real data. The work also emphasizes reproducibility, anti-misuse safeguards, and ethical considerations, while acknowledging higher computational costs as a limitation and future direction for efficiency.

Abstract

With recent advancements in visual synthesis, there is a growing risk of encountering images with detrimental effects, such as hate, discrimination, or privacy violations. The research on transforming harmful images into responsible ones remains unexplored. In this paper, we formulate a new task, responsible visual editing, which entails modifying specific concepts within an image to render it more responsible while minimizing changes. However, the concept that needs to be edited is often abstract, making it challenging to locate what needs to be modified and plan how to modify it. To tackle these challenges, we propose a Cognitive Editor (CoEditor) that harnesses the large multimodal model through a two-stage cognitive process: (1) a perceptual cognitive process to focus on what needs to be modified and (2) a behavioral cognitive process to strategize how to modify. To mitigate the negative implications of harmful images on research, we create a transparent and public dataset, AltBear, which expresses harmful information using teddy bears instead of humans. Experiments demonstrate that CoEditor can effectively comprehend abstract concepts within complex scenes and significantly surpass the performance of baseline models for responsible visual editing. We find that the AltBear dataset corresponds well to the harmful content found in real images, offering a consistent experimental evaluation, thereby providing a safer benchmark for future research. Moreover, CoEditor also shows great results in general editing. We release our code and dataset at https://github.com/kodenii/Responsible-Visual-Editing.

Figures (12)

• Figure 1: Overview of responsible visual editing. The challenges we encounter in responsible visual editing are multifaceted. Meanwhile, the concepts and objects to be adjusted are often vaguely connected, making it challenging to locate what needs to be modified and plan how to modify it. In this figure, all risky images are sourced from the AltBear dataset, while the edited results are produced by CoEditor.
• Figure 2: Overview of CoEditor. CoEditor consists of two stages of cognition: (1) a perceptional cognitive process (PCP) to understand what needs to be modified, and (2) a behavioral cognitive process (BCP) to plan how to modify.
• Figure 3: Overall visualized results of AltBear. CoEditor perform well in all subtasks and maintained high visual similarity. Not only that, we also find that the results of CoEditor have stronger rationality. InstructDiffusion often tends to over-edit images or produce unreasonable visual effects, while the editing ability of InstructPix2pix is weaker compared to CoEditor.
• Figure 4: Results in general editing. CoEditor perform well even in general editing with keeping background unchanged.
• Figure 5: Results of different components in CoEditor. Without the help of BCP, the CoEditor produce results inconsistent with the concept or visually unreasonable because it does not know how to modify the content correctly. Without PCP, CoEditor is unable to locate the editing regions, resulting in editing failure.