MonetGPT: Solving Puzzles Enhances MLLMs' Image Retouching Skills

TL;DR

MonetGPT advances procedural, non-destructive image retouching by fine-tuning a multimodal LLM to become operation-aware, capable of critiquing a source image, planning a sequence of edits from a fixed operation library, and executing those edits with interpretable reasoning. The authors introduce three visual puzzles to ground the model in individual operations, aesthetics, and editing plans, and they synthesize a reasoning dataset to supervise fine-tuning. The approach yields an interpretable, high-resolution, parameterized editing pipeline that outperforms open baselines and rivals Google Photos on key metrics, while avoiding identity loss common to generative edits. This framework offers a flexible, user-overridable, and explainable pathway for professional-quality retouching that can be extended with broader operation sets and user preferences.

Abstract

Retouching is an essential task in post-manipulation of raw photographs. Generative editing, guided by text or strokes, provides a new tool accessible to users but can easily change the identity of the original objects in unacceptable and unpredictable ways. In contrast, although traditional procedural edits, as commonly supported by photoediting tools (e.g., Gimp, Lightroom), are conservative, they are still preferred by professionals. Unfortunately, professional quality retouching involves many individual procedural editing operations that is challenging to plan for most novices. In this paper, we ask if a multimodal large language model (MLLM) can be taught to critique raw photographs, suggest suitable remedies, and finally realize them with a given set of pre-authored procedural image operations. We demonstrate that MLLMs can be first made aware of the underlying image processing operations, by training them to solve specially designed visual puzzles. Subsequently, such an operation-aware MLLM can both plan and propose edit sequences. To facilitate training, given a set of expert-edited photos, we synthesize a reasoning dataset by procedurally manipulating the expert edits and then grounding a pretrained LLM on the visual adjustments, to synthesize reasoning for finetuning. The proposed retouching operations are, by construction, understandable by the users, preserve object details and resolution, and can be optionally overridden. We evaluate our setup on a variety of test examples and show advantages, in terms of explainability and identity preservation, over existing generative and other procedural alternatives. Code, data, models, and supplementary results can be found via our project website at https://monetgpt.github.io.

Figures (9)

• Figure 1: Generative tools, for example instructPix2Pix brooks2022instructpix2pix or MGIE mgie24, produce impressive image enhancements but can result in identity loss (e.g., faces, hands, objects) and are harder to override by users. Procedural approaches are more controllable as they restrict operations to a given set of user-prescribed operation library, and can be overridden or applied in parts. Current MLLMs (bottom-left: e.g., GPT o1 applied using a library of operations, presented as doc-strings), do not have a good internal model of image operations, and perform worse than our operation-aware variant (bottom-right). See \ref{['sec:evaluation']} for evaluation.
• Figure 2: Puzzle A. This puzzle intends to teach what any single operation $\mathcal{O}$$\in$$\mathcal{L}$, along with its value $V$, does to a source image $I_S$ to produce an edited image $I_E$. The visual puzzle being, given an ordered pair $(I_S, I_E)$, one has to predict $(\mathcal{O},V)$. Using $(I_S, I_E)$, we also generate corresponding reasoning $R$.
• Figure 3: Puzzle B. This puzzle intends to teach about image aesthetics under any single operation $\mathcal{O}$$\in$$\mathcal{L}$. The visual puzzle being, given a set of randomly ordered images $(I_E, I_{V_1}, I_{V_2}, I_{V_3}, I_{V_4})$, generated from an expert-edited final image $I_X$ by applying operation $\mathcal{O}$ with perturbed values $\{V_i\}$, one has to order the set of images, based on low to high values $V_i$, as well as identify the optimal image $I_X$ along with the perturbations values to go back to $I_X$ from each image. Note that we implicitly assume that any perturbation of an expert-edited image results in a worse image. Using the image set and the operation, we also generate corresponding reasoning $R$.
• Figure 4: Puzzle C. This puzzle intends to teach how to generate a retouching plan. The visual puzzle being, given an ordered pair $(I_S, I_E)$, one has to come up with a retouching plan $\{(\mathcal{O}_i, V_i)\}$ listing the operations, from $\mathcal{L}$, along with the associated parameter values. Using the image retouching sequence and the operations, we also generate corresponding reasoning $R$, in the form of <Adjustment, Issue, Solution> for each operation.
• Figure 5: Each row depicts an input image and a retouched version produced by each of the baselines. While generative baselines (MGIE) struggle to preserve the identity (last row), Exposure or Gemini sometimes generate too bright or dark results (third row). Direct regression with an MLLM fails to make sufficient enhancements. Our method is capable of providing balanced and aesthetic enhancements.