EraseLoRA: MLLM-Driven Foreground Exclusion and Background Subtype Aggregation for Dataset-Free Object Removal

TL;DR

EraseLoRA introduces a dataset-free object removal framework that leverages MLLM-guided background-aware reasoning (BFE) and test-time diffusion reconstruction with background subtype aggregation (BRSA). By explicitly separating target foreground, non-target foregrounds, and background, and by aggregating multiple background cues through a BRSA objective, it avoids the artifacts of attention-surgery methods while preserving fine detail and global structure. Across OpenImages V7 and RORD benchmarks, EraseLoRA yields substantial gains in Background Similarity and strong suppression of foreground regeneration, outperforming prior dataset-free methods and remaining competitive with dataset-driven approaches. As a plug-in, model-agnostic approach, it enables faithful background reconstruction without requiring paired data or retraining, offering a practical path for robust, background-consistent object removal.

Abstract

Object removal differs from common inpainting, since it must prevent the masked target from reappearing and reconstruct the occluded background with structural and contextual fidelity, rather than merely filling a hole plausibly. Recent dataset-free approaches that redirect self-attention inside the mask fail in two ways: non-target foregrounds are often misinterpreted as background, which regenerates unwanted objects, and direct attention manipulation disrupts fine details and hinders coherent integration of background cues. We propose EraseLoRA, a novel dataset-free framework that replaces attention surgery with background-aware reasoning and test-time adaptation. First, Background-aware Foreground Exclusion (BFE), uses a multimodal large-language models to separate target foreground, non-target foregrounds, and clean background from a single image-mask pair without paired supervision, producing reliable background cues while excluding distractors. Second, Background-aware Reconstruction with Subtype Aggregation (BRSA), performs test-time optimization that treats inferred background subtypes as complementary pieces and enforces their consistent integration through reconstruction and alignment objectives, preserving local detail and global structure without explicit attention intervention. We validate EraseLoRA as a plug-in to pretrained diffusion models and across benchmarks for object removal, demonstrating consistent improvements over dataset-free baselines and competitive results against dataset-driven methods. The code will be made available upon publication.

Figures (25)

• Figure 1: Qualitative comparison with prior dataset-free methods. Previous state-of-the-art approaches jia2025designeditsun2025attentiveeraser treat only the masked region as foreground, misinterpreting non-target objects as background and regenerating them. EraseLoRA identifies and excludes non-target foregrounds and reconstructs the masked region using background cues, enabling faithful object removal.
• Figure 2: Artifacts from attention manipulation. Recent dataset-free methods jia2025designeditsun2025attentiveeraser directly modify attention inside the mask, leading to blurred or distorted background textures, whereas EraseLoRA aggregates background subtypes without attention blocking and preserves sharp, coherent structures.
• Figure 3: Background-aware reasoning power of MLLM. Unlike prior works kim2025chainofzoomwang2024genartistQu2025silmmzhou2025fireedit employ MLLMs for visual reasoning over the visible scene, we first leverage MLLMs to infer background cues behind the masked target.
• Figure 4: Overview of EraseLoRA. BFE (\ref{['sec:bfe']}) separates target foreground, non-target foregrounds, and background from a single image-mask pair using an MLLM zhu2025internvl3 and Tag2Mask models liu2024groundingdinoravi2025sam2. After producing clean background cues, BRSA (\ref{['sec:brsa']}) performs test-time adaptation wang2020tta with reconstruction and alignment objectives, coherently integrating background subtypes into the masked region.
• Figure 5: Identification of non-target foregrounds. Prior methods chen2024freecomposesun2025attentiveeraser treat the entire unmasked region as background, which causes regeneration of non-target foregrounds. In contrast, EraseLoRA explicitly identifies non-target foregrounds within the mask and excludes them, producing clean background.