Example-Based Feature Painting on Textures

TL;DR

This work tackles automatic painting of non-stationary texture features from small unlabeled image sets. It introduces a three-stage pipeline combining unsupervised anomaly localization, pixel-level feature clustering via contrastive learning, and spatially conditioned diffusion-based texture generation, with noise-mixing for editing and noise-uniformization for large textures. The method enables interactive painting, feature transfer, and tileable textures, demonstrated on multiple texture datasets including MVTec textures and SVBRDF synthesis. It achieves accurate pixel-level segmentation of feature types and enables high-quality editing and large-scale texture synthesis with minimal per-edit latency.

Abstract

In this work, we propose a system that covers the complete workflow for achieving controlled authoring and editing of textures that present distinctive local characteristics. These include various effects that change the surface appearance of materials, such as stains, tears, holes, abrasions, discoloration, and more. Such alterations are ubiquitous in nature, and including them in the synthesis process is crucial for generating realistic textures. We introduce a novel approach for creating textures with such blemishes, adopting a learning-based approach that leverages unlabeled examples. Our approach does not require manual annotations by the user; instead, it detects the appearance-altering features through unsupervised anomaly detection. The various textural features are then automatically clustered into semantically coherent groups, which are used to guide the conditional generation of images. Our pipeline as a whole goes from a small image collection to a versatile generative model that enables the user to interactively create and paint features on textures of arbitrary size. Notably, the algorithms we introduce for diffusion-based editing and infinite stationary texture generation are generic and should prove useful in other contexts as well. Project page: https://reality.tf.fau.de/pub/ardelean2025examplebased.html

Figures (25)

• Figure 1: Feature-conditioned results generated by our model on various textures. The label maps are sketched by the user/artist. Please note the graceful, realistic transitions between the normal class and the painted features; digital zoom recommended.
• Figure 2: Our 3-stage pipeline for obtaining a generative model for textures with prominent features.
• Figure 3: Results of our pixel-level anomaly segmentation. We seek to have the background pixels mapped to a single class and the different features to distinct labels. The figure compares the results of our approach, BlindLCA ardelean2024blind, and STEGO hamilton2022unsupervised.
• Figure 4: Image editing visualization, showing: the original image, irregular feature examples from the dataset, desired edit mask, the image after being encoded and decoded by the VAE of SD, the result of our diffusion model for the inverted noise, and the result of our noise-mixing synthesis.
• Figure 5: Editing results with our noise-mixing approach. The rows depict top to bottom: a real photograph, the conditioning label-map, the result obtained with TextureReformer wang2022texture, and our edit.