Content-aware Tile Generation using Exterior Boundary Inpainting

Abstract

We present a novel and flexible learning-based method for generating tileable image sets. Our method goes beyond simple self-tiling, supporting sets of mutually tileable images that exhibit a high degree of diversity. To promote diversity we decouple structure from content by foregoing explicit copying of patches from an exemplar image. Instead we leverage the prior knowledge of natural images and textures embedded in large-scale pretrained diffusion models to guide tile generation constrained by exterior boundary conditions and a text prompt to specify the content. By carefully designing and selecting the exterior boundary conditions, we can reformulate the tile generation process as an inpainting problem, allowing us to directly employ existing diffusion-based inpainting models without the need to retrain a model on a custom training set. We demonstrate the flexibility and efficacy of our content-aware tile generation method on different tiling schemes, such as Wang tiles, from only a text prompt. Furthermore, we introduce a novel Dual Wang tiling scheme that provides greater texture continuity and diversity than existing Wang tile variants.

Figures (17)

• Figure 1: Self-tiling Texture Generation: We establish contiguous horizontal and vertical boundary conditions by selecting two template patches from the exemplar, and cutting them in half horizontally and vertically respectively. Each cut half is placed on the outside of the tile with the cut edge (dashed line) aligned with a tile edge. The interior of the tile (scribbled area) is then inpainted. The final tile is then cropped to only retain the synthesized part.
• Figure 2: Comparison of Tiling Schemes: A comparison of the different tiling schemes (first row: self-tiling and stochastic self-tiling (with $4$ tiles); second row: stochastic Escher self-tiling (with $4$ tiles) and regular $3$-color Wang tiling ($81$ tiles); last row: $3$-color Dual Wang tiling) demonstrated on texture tiles generated with the prompt "European city blocks, tile roofs, streams, drone footage". For each example we also show the tile shape and size with the scribbled overlay.
• Figure 3: Wang Tile Generation: Given $C$ colors, we select $2 \times C$ template patches from an exemplar image, and cut each template patch in half (a horizontal and a vertical cut per color, marked by the dashed line). For each Wang tile we set the boundary conditions by coping each half template patch to the exterior of the Wang tile with the (dashed) edge matching the corresponding Wang tile edge. Finally, we generate the Wang tile texture by inpainting the interior region (i.e., the scribbled area).
• Figure 4: Corner Wang Generation: We follow a very similar generation process as Lagae and Dutré Lagae:2005:POD. Instead of using square template patches, we select $C$ diamond shaped template patches, and cut them both horizontally and vertically. Next, we copy the triangular patches into the corner tiles based on corner colors, and inpaint the interior. Unlike the other tiling schemes, Corner Wang tiles include (parts of) the template patches in the final tiled texture.
• Figure 5: Content-aware Corner Wang Tiling: An example of a Corner Wang tiling Lagae:2005:POD generated with an adaptation of our content-aware tile synthesis method. Unlike regular Wang tiles and Dual Corner Wang tiles, the resulting tiles contain verbatim copied texels from the exemplar image, resulting in a less diverse tiling.