Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

SIRR-LMM: Single-image Reflection Removal via Large Multimodal Model

Yu Guo, Zhiqiang Lao, Xiyun Song, Yubin Zhou, Heather Yu

TL;DR

This paper tackles the ill-posed problem of single-image reflection removal by building a physically realistic synthetic dataset that renders path-traced glass over real backgrounds using HDR lighting. It then fine-tunes a Large Multimodal Model (LMM) (Flux.1 Kontext) with Low-Rank Adaptation (LoRA) on a concatenated [I:T:R] input and a unified prompt, achieving state-of-the-art-like performance with relatively small synthetic data. Key contributions include the physically grounded data generation framework, the in-context LoRA approach for SIRR, and extensive quantitative and qualitative validation across real-world benchmarks, user studies, and ablation analyses. The work demonstrates the practical impact of realistic data and task-specific LMM fine-tuning for robust reflection removal and separation in diverse real-world scenes.

Abstract

Glass surfaces create complex interactions of reflected and transmitted light, making single-image reflection removal (SIRR) challenging. Existing datasets suffer from limited physical realism in synthetic data or insufficient scale in real captures. We introduce a synthetic dataset generation framework that path-traces 3D glass models over real background imagery to create physically accurate reflection scenarios with varied glass properties, camera settings, and post-processing effects. To leverage the capabilities of Large Multimodal Model (LMM), we concatenate the image layers into a single composite input, apply joint captioning, and fine-tune the model using task-specific LoRA rather than full-parameter training. This enables our approach to achieve improved reflection removal and separation performance compared to state-of-the-art methods.

SIRR-LMM: Single-image Reflection Removal via Large Multimodal Model

TL;DR

This paper tackles the ill-posed problem of single-image reflection removal by building a physically realistic synthetic dataset that renders path-traced glass over real backgrounds using HDR lighting. It then fine-tunes a Large Multimodal Model (LMM) (Flux.1 Kontext) with Low-Rank Adaptation (LoRA) on a concatenated [I:T:R] input and a unified prompt, achieving state-of-the-art-like performance with relatively small synthetic data. Key contributions include the physically grounded data generation framework, the in-context LoRA approach for SIRR, and extensive quantitative and qualitative validation across real-world benchmarks, user studies, and ablation analyses. The work demonstrates the practical impact of realistic data and task-specific LMM fine-tuning for robust reflection removal and separation in diverse real-world scenes.

Abstract

Glass surfaces create complex interactions of reflected and transmitted light, making single-image reflection removal (SIRR) challenging. Existing datasets suffer from limited physical realism in synthetic data or insufficient scale in real captures. We introduce a synthetic dataset generation framework that path-traces 3D glass models over real background imagery to create physically accurate reflection scenarios with varied glass properties, camera settings, and post-processing effects. To leverage the capabilities of Large Multimodal Model (LMM), we concatenate the image layers into a single composite input, apply joint captioning, and fine-tune the model using task-specific LoRA rather than full-parameter training. This enables our approach to achieve improved reflection removal and separation performance compared to state-of-the-art methods.
Paper Structure (23 sections, 4 equations, 14 figures, 4 tables)

This paper contains 23 sections, 4 equations, 14 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Reflection Removal and Separation
  4. Existing Datasets
  5. Image synthesis LMM conditioned on text
  6. Data Generation
  7. Path-traced scene setup
  8. Scene parameter variation
  9. Text-to-image Model Fine-tuning
  10. Experimental Setup
  11. Results and Analysis
  12. Quantitative Performance
  13. Qualitative Comparison
  14. Single-image reflection removal
  15. Single-image reflection separation
  16. ...and 8 more sections

Figures (14)

  • Figure 1: Reflection and transmission. (a) The I nput image with glass reflection; (b) The T ransmission component of I; (c) The B ackground image without glass; (d) The R eflection component of I; (e) The Mirror Reflection, which is considered as back scene image without any glass effects. We use blue tint glass to better illustrate the difference between T and B.
  • Figure 2: Three different scenes setup. (a) Use HDR environment maps for both transmission and reflection. (b) Use HDR map for transmission and put a LDR image behind camera for reflection. (c) Use HDR map for reflection and put a LDR image behind the glass for transmission.
  • Figure 3: Blurry effects. (a) and (b): different roughness of the glass surface. (c) and (d): different aperture sizes of the camera.
  • Figure 4: Double reflection illustration. (a) Transmission: turn off the reflection for both glass layers. (b) Reflection: only turn off the transmission for the bottom glass. (c) An example of double reflection. (d) A compared single reflection.
  • Figure 5: Fine-tuning pipeline. A overview of LoRA fine-tuning FLUX.1 Kontext with consolidated image pairs ([I:T:R]) and consolidated prompts.
  • ...and 9 more figures