ReasonX: MLLM-Guided Intrinsic Image Decomposition

TL;DR

This work tackles the generalization gap in intrinsic image decomposition when trained primarily on synthetic data. It introduces ReasonX, which uses a frozen MLLM judge to produce relative intrinsic judgments from RGB images and employs GRPO to fine-tune base intrinsic predictors on unlabeled real images. By injecting exploration and leveraging group-based rewards, ReasonX aligns judge judgments with analytic relations derived from predicted intrinsics, yielding improvements in albedo, depth, and normals across real-world datasets. The approach achieves 9-25% WHDR reductions on IIW and up to 46% depth accuracy gains on ETH3D, demonstrating strong cross-domain generalization and cross-modal consistency. ReasonX is model-agnostic and highlights MLLM-guided comparative supervision as a general paradigm for cross-modal inverse rendering tasks.

Abstract

Intrinsic image decomposition aims to separate images into physical components such as albedo, depth, normals, and illumination. While recent diffusion- and transformer-based models benefit from paired supervision from synthetic datasets, their generalization to diverse, real-world scenarios remains challenging. We propose ReasonX, a novel framework that leverages a multimodal large language model (MLLM) as a perceptual judge providing relative intrinsic comparisons, and uses these comparisons as GRPO rewards for fine-tuning intrinsic decomposition models on unlabeled, in-the-wild images. Unlike RL methods for generative models, our framework aligns conditional intrinsic predictors by rewarding agreement between the judge's relational assessments and analytically derived relations from the model's outputs. ReasonX is model-agnostic and can be applied to different intrinsic predictors. Across multiple base architectures and modalities, ReasonX yields significant improvements, including 9-25% WHDR reduction on IIW albedo and up to 46% depth accuracy gains on ETH3D, highlighting the promise of MLLM-guided comparative supervision to bridge low- and high-level vision reasoning.

Figures (11)

• Figure 1: We propose ReasonX, a novel framework for MLLM-guided improvement of intrinsic decomposition models via relative intrinsic judgments on RGB input images.
• Figure 2: Overview of our ReasonX framework. (a) We fine-tune an MLLM to judge relative intrinsic properties from RGB images using sampled point pairs. (b) The frozen judge then provides rewards within a GRPO loop to refine an intrinsic decomposition model $\pi$: for each RGB image, we generate a group of $G=8$ samples, query the judge across point pairs and modalities, and compute group-relative rewards to update $\pi$ without ground-truth intrinsics.
• Figure 3: Intrinsic decomposition samples on challenging out-of-distribution images. Our PRISM-X significantly improves its base model PRISM across all intrinsic channels with respect to decomposition quality and in-the-wild generalization performance.
• Figure 4: Our MLLM judge is trained on synthetic data to make relative intrinsic judgments from point-pair annotated RGB images and modality-specific questions. Ground-truth answers are derived from the corresponding intrinsic maps.
• Figure 5: Depth estimation comparisons of our PRISM-X with its base model PRISM and SOTA baseline method Marigold Depth v1.0 on samples from the NYUv2, DIODE and ETH3D datasets. PRISM-X performs significantly better on challenging images.