Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

CLIPtone: Unsupervised Learning for Text-based Image Tone Adjustment

Hyeongmin Lee, Kyoungkook Kang, Jungseul Ok, Sunghyun Cho

TL;DR

CLIPtone introduces an unsupervised framework for text-guided image tone adjustment that leverages CLIP as a perceptual prior to align tonal edits with natural language descriptions. It builds a text adapter that, via a hyper-network, modulates a backbone 3D-LUT-based tone adjustment network by adjusting the AdaInt and weight-predictor modules, enabling diverse, zero-shot descriptions without paired training data. Training relies on two unpaired streams (Color Names with appended 'photo' and MIT-Adobe 5K source images) and a composite loss consisting of content preservation, CLIP directional alignment, and LUT regularization, including a sampling-interval term to ensure smooth color transitions. Across qualitative and quantitative experiments, CLIPtone outperforms baselines in preserving image structure while achieving text-aligned tonal changes, and it demonstrates efficiency and zero-shot generalization, highlighting its practical impact for light, flexible image editing with natural language guidance.

Abstract

Recent image tone adjustment (or enhancement) approaches have predominantly adopted supervised learning for learning human-centric perceptual assessment. However, these approaches are constrained by intrinsic challenges of supervised learning. Primarily, the requirement for expertly-curated or retouched images escalates the data acquisition expenses. Moreover, their coverage of target style is confined to stylistic variants inferred from the training data. To surmount the above challenges, we propose an unsupervised learning-based approach for text-based image tone adjustment method, CLIPtone, that extends an existing image enhancement method to accommodate natural language descriptions. Specifically, we design a hyper-network to adaptively modulate the pretrained parameters of the backbone model based on text description. To assess whether the adjusted image aligns with the text description without ground truth image, we utilize CLIP, which is trained on a vast set of language-image pairs and thus encompasses knowledge of human perception. The major advantages of our approach are three fold: (i) minimal data collection expenses, (ii) support for a range of adjustments, and (iii) the ability to handle novel text descriptions unseen in training. Our approach's efficacy is demonstrated through comprehensive experiments, including a user study.

CLIPtone: Unsupervised Learning for Text-based Image Tone Adjustment

TL;DR

CLIPtone introduces an unsupervised framework for text-guided image tone adjustment that leverages CLIP as a perceptual prior to align tonal edits with natural language descriptions. It builds a text adapter that, via a hyper-network, modulates a backbone 3D-LUT-based tone adjustment network by adjusting the AdaInt and weight-predictor modules, enabling diverse, zero-shot descriptions without paired training data. Training relies on two unpaired streams (Color Names with appended 'photo' and MIT-Adobe 5K source images) and a composite loss consisting of content preservation, CLIP directional alignment, and LUT regularization, including a sampling-interval term to ensure smooth color transitions. Across qualitative and quantitative experiments, CLIPtone outperforms baselines in preserving image structure while achieving text-aligned tonal changes, and it demonstrates efficiency and zero-shot generalization, highlighting its practical impact for light, flexible image editing with natural language guidance.

Abstract

Recent image tone adjustment (or enhancement) approaches have predominantly adopted supervised learning for learning human-centric perceptual assessment. However, these approaches are constrained by intrinsic challenges of supervised learning. Primarily, the requirement for expertly-curated or retouched images escalates the data acquisition expenses. Moreover, their coverage of target style is confined to stylistic variants inferred from the training data. To surmount the above challenges, we propose an unsupervised learning-based approach for text-based image tone adjustment method, CLIPtone, that extends an existing image enhancement method to accommodate natural language descriptions. Specifically, we design a hyper-network to adaptively modulate the pretrained parameters of the backbone model based on text description. To assess whether the adjusted image aligns with the text description without ground truth image, we utilize CLIP, which is trained on a vast set of language-image pairs and thus encompasses knowledge of human perception. The major advantages of our approach are three fold: (i) minimal data collection expenses, (ii) support for a range of adjustments, and (iii) the ability to handle novel text descriptions unseen in training. Our approach's efficacy is demonstrated through comprehensive experiments, including a user study.
Paper Structure (26 sections, 5 equations, 9 figures, 4 tables)

This paper contains 26 sections, 5 equations, 9 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Image Tone Adjustment Methods.
  4. Text-driven Generative Models.
  5. CLIP as a Perceptual Criterion
  6. Network Architecture of CLIPtone
  7. Tone Adjustment Network.
  8. Text Adapter.
  9. Unsupervised Learning for CLIPtone.
  10. Training Strategy.
  11. Training Loss.
  12. Experiments
  13. Comparative Evaluation
  14. Baselines.
  15. Qualitative Comparisons.
  16. ...and 11 more sections

Figures (9)

  • Figure 1: We present CLIPtone, a text-based image tone adjustment framework trained in an unsupervised manner. With its superior understanding of natural languages, CLIPtone is capable of performing successful adjustments across a range of text descriptions, including those previously deemed challenging.
  • Figure 2: We apply tone adjustment filters from Adobe Lightroom Classic adobelightroomclassic to 500 images from the MIT-Adobe 5K dataset ma5k, and calculate the relative similarities between the images and filter names in the CLIP space. For all filters, the filtered images have higher similarity scores than the source images, implying that CLIP can assess the tonal properties of images in a manner aligning with human perception.
  • Figure 3: CLIPtone consists of a text adapter and a tone adjustment network. From a target text description, the text adapter calculates a directional vector within the CLIP embedding space from the source to target text descriptions and estimates the modulation parameter $\Delta\theta$ for the AdaInt module and the weight predictor of the tone adjustment network. The modulated tone adjustment network adaptively constructs an image-text adaptive 3D LUT through fusing basis 3D LUTs and non-uniform sampling, ultimately adjusting the color values of an input image.
  • Figure 4: Qualitative comparisons against baselines for modifications of input images according to given text descriptions. T2ONet t2onet induces only subtle changes and fails to perform appropriate adjustments. While CLIPstyler kwon2022clipstyler and IP2P brooks2023instructpix2pix make appropriate adjustments aligned with the text descriptions, they fail to preserve the image contents. In contrast, CLIPtone successfully makes appropriate adjustments while preserving the contents of the input images.
  • Figure 5: Qualitative comparison against pseudo-ground-truth images created by tone adjustment filters of Adobe Lightroom Classic adobelightroomclassic. Compared to the baselines, CLIPtone performs more similar adjustment to the Adobe filters while preserving the original image structure.
  • ...and 4 more figures