Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Interaction-Consistent Object Removal via MLLM-Based Reasoning

Ching-Kai Huang, Wen-Chieh Lin, Yan-Cen Lee

TL;DR

This work introduces Interaction-Consistent Object Removal (ICOR), which requires removing a target object along with all interacting scene elements to preserve semantic coherence. It proposes REORM, a modular framework that uses multimodal large language models (MLLMs) for commonsense reasoning to identify associated elements, combined with mask-guided removal and a self-correction loop to ensure consistency. The authors introduce ICOREval, a benchmark of instruction-driven removal cases with rich interaction dependencies. Experimental results show that REORM outperforms state-of-the-art language-guided editing methods in image quality and interaction consistency, and a local-deployment variant demonstrates resource-efficient, privacy-preserving editing on limited hardware.

Abstract

Image-based object removal often erases only the named target, leaving behind interaction evidence that renders the result semantically inconsistent. We formalize this problem as Interaction-Consistent Object Removal (ICOR), which requires removing not only the target object but also associated interaction elements, such as lighting-dependent effects, physically connected objects, targetproduced elements, and contextually linked objects. To address this task, we propose Reasoning-Enhanced Object Removal with MLLM (REORM), a reasoningenhanced object removal framework that leverages multimodal large language models to infer which elements must be jointly removed. REORM features a modular design that integrates MLLM-driven analysis, mask-guided removal, and a self-correction mechanism, along with a local-deployment variant that supports accurate editing under limited resources. To support evaluation, we introduce ICOREval, a benchmark consisting of instruction-driven removals with rich interaction dependencies. On ICOREval, REORM outperforms state-of-the-art image editing systems, demonstrating its effectiveness in producing interactionconsistent results.

Interaction-Consistent Object Removal via MLLM-Based Reasoning

TL;DR

This work introduces Interaction-Consistent Object Removal (ICOR), which requires removing a target object along with all interacting scene elements to preserve semantic coherence. It proposes REORM, a modular framework that uses multimodal large language models (MLLMs) for commonsense reasoning to identify associated elements, combined with mask-guided removal and a self-correction loop to ensure consistency. The authors introduce ICOREval, a benchmark of instruction-driven removal cases with rich interaction dependencies. Experimental results show that REORM outperforms state-of-the-art language-guided editing methods in image quality and interaction consistency, and a local-deployment variant demonstrates resource-efficient, privacy-preserving editing on limited hardware.

Abstract

Image-based object removal often erases only the named target, leaving behind interaction evidence that renders the result semantically inconsistent. We formalize this problem as Interaction-Consistent Object Removal (ICOR), which requires removing not only the target object but also associated interaction elements, such as lighting-dependent effects, physically connected objects, targetproduced elements, and contextually linked objects. To address this task, we propose Reasoning-Enhanced Object Removal with MLLM (REORM), a reasoningenhanced object removal framework that leverages multimodal large language models to infer which elements must be jointly removed. REORM features a modular design that integrates MLLM-driven analysis, mask-guided removal, and a self-correction mechanism, along with a local-deployment variant that supports accurate editing under limited resources. To support evaluation, we introduce ICOREval, a benchmark consisting of instruction-driven removals with rich interaction dependencies. On ICOREval, REORM outperforms state-of-the-art image editing systems, demonstrating its effectiveness in producing interactionconsistent results.
Paper Structure (20 sections, 10 figures, 6 tables)

This paper contains 20 sections, 10 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Task Overview: Interaction-Consistent Object Removal
  4. Method: Reasoning-Enhanced Object Removal with MLLM
  5. MLLM-driven Analysis and Removal
  6. MLLM-controlled Self-Correction
  7. MLLM–LLM Collaboration for Local Deployment
  8. Experiments
  9. Experimental Setup
  10. Comparison with Language-guided Image Editing Methods
  11. Verification of Local Deployment Design
  12. Failure Case Analysis
  13. Discussion: Toward Intent-Aware and Privacy-Preserving Object Removal
  14. Conclusion
  15. Appendix: Additional Results
  16. ...and 5 more sections

Figures (10)

  • Figure 1: We introduce the task of Interaction-Consistent Object Removal (ICOR), which aims to remove not only the target object itself but also its associated interaction elements, in order to produce semantically coherent and visually realistic outputs. These interaction effects can be broadly categorized into four types: lighting-dependent effects, physically connected objects, target-produced elements, and contextually linked objects.
  • Figure 2: Overview of the proposed REORM, which leverages the commonsense reasoning capabilities of an MLLM to enable interaction-consistent object removal.
  • Figure 3: Overview of REORM for local deployment, which adopts prompt chaining strategy and coordinates a compact MLLM with an LLM.
  • Figure 4: Qualitative comparison on ICOREval. Our method achieves ICOR, successfully removing a rider with the bicycle, a person with a cast shadow, a person holding a watering can, and a cat playing with a toy and its string.
  • Figure 5: Qualitative comparison on images without ground truth. REORM removes not only target objects but also associated elements such as shadows, footprints, watering can and streams, and signage, with the GPT-4o variant showing stronger generalization in challenging cases.
  • ...and 5 more figures