RAVE: Residual Vector Embedding for CLIP-Guided Backlit Image Enhancement

TL;DR

This paper tackles backlit image enhancement under unpaired data by improving CLIP-guided guidance. It introduces two latent-space strategies: CLIP-LIT-Latent, which learns guidance vectors directly in the CLIP latent space without a text encoder, and RAVE, which uses a fixed residual vector v_residual derived from mean CLIP embeddings to steer the enhancement network in a single, stable training stage. Both methods maintain lightweight inference and support training with paired or unpaired data, achieving competitive or superior results and reducing training time significantly (up to ~25×). Importantly, RAVE offers interpretability of the guidance vector, enabling bias analysis and potential correction in training data, with qualitative improvements such as reduced artifacts and better contrast. The work thus advances CLIP-guided backlit enhancement by delivering faster, more robust training and enabling bias-aware improvements over prior approaches like CLIP-LIT and Diff-Retinex.

Abstract

In this paper we propose a novel modification of Contrastive Language-Image Pre-Training (CLIP) guidance for the task of unsupervised backlit image enhancement. Our work builds on the state-of-the-art CLIP-LIT approach, which learns a prompt pair by constraining the text-image similarity between a prompt (negative/positive sample) and a corresponding image (backlit image/well-lit image) in the CLIP embedding space. Learned prompts then guide an image enhancement network. Based on the CLIP-LIT framework, we propose two novel methods for CLIP guidance. First, we show that instead of tuning prompts in the space of text embeddings, it is possible to directly tune their embeddings in the latent space without any loss in quality. This accelerates training and potentially enables the use of additional encoders that do not have a text encoder. Second, we propose a novel approach that does not require any prompt tuning. Instead, based on CLIP embeddings of backlit and well-lit images from training data, we compute the residual vector in the embedding space as a simple difference between the mean embeddings of the well-lit and backlit images. This vector then guides the enhancement network during training, pushing a backlit image towards the space of well-lit images. This approach further dramatically reduces training time, stabilizes training and produces high quality enhanced images without artifacts, both in supervised and unsupervised training regimes. Additionally, we show that residual vectors can be interpreted, revealing biases in training data, and thereby enabling potential bias correction.

Figures (4)

• Figure 1: Visual comparison of results obtained by RAVE vs CLIP-LIT. RAVE produces well-lit images with fewer artifacts (note the dark green color of underexposed regions in CLIP-LIT), as shown in the zoomed-in sections. Further comparisons can be found in supplementary material.
• Figure 2: Overview of the original CLIP-LIT approach and proposed CLIP-LIT-Latent. (a) depicts the first stage of training, which consists of prompt or latent vector initialization and the initial training of an enhancement network. (b) shows the second stage, where prompt/latent vector refinement and enhancement model fine-tuning are iteratively repeated. Blue and red boxes are related to CLIP-LIT and CLIP-LIT-Latent.
• Figure 3: Overview of the RAVE model. (a) First, we calculate residual vector $\mathbf{v}_{\text{residual}}$ based on backlit and well-lit training data. (b) Then we switch to enhancement model training based on the identity loss and the loss based on the residual vector.
• Figure 4: Visual comparison of results by CLIP-LIT, CLIP-LIT-Latent and RAVE.