Generative Escher Meshes

TL;DR

A differentiable family of linear systems derived from a 2D mesh-mapping technique - Orbifold Tutte Embedding - is constructed by considering the mesh’s Laplacian matrix as differentiable parameters, thereby providing an end-to-end differentiable representation for the entire space of valid tiles.

Abstract

This paper proposes a fully-automatic, text-guided generative method for producing perfectly-repeating, periodic, tile-able 2D imagery, such as the one seen on floors, mosaics, ceramics, and the work of M.C. Escher. In contrast to square texture images that are seamless when tiled, our method generates non-square tilings which comprise solely of repeating copies of the same object. It achieves this by optimizing both geometry and texture of a 2D mesh, yielding a non-square tile in the shape and appearance of the desired object, with close to no additional background details, that can tile the plane without gaps nor overlaps. We enable optimization of the tile's shape by an unconstrained, differentiable parameterization of the space of all valid tileable meshes for given boundary conditions stemming from a symmetry group. Namely, we construct a differentiable family of linear systems derived from a 2D mesh-mapping technique - Orbifold Tutte Embedding - by considering the mesh's Laplacian matrix as differentiable parameters. We prove that the solution space of these linear systems is exactly all possible valid tiling configurations, thereby providing an end-to-end differentiable representation for the entire space of valid tiles. We render the textured mesh via a differentiable renderer, and leverage a pre-trained image diffusion model to induce a loss on the resulting image, updating the mesh's parameters so as to make its appearance match the text prompt. We show our method is able to produce plausible, appealing results, with non-trivial tiles, for a variety of different periodic tiling patterns.

Figures (25)

• Figure 1: The other "interesting" wallpaper groups left out of Figure \ref{['fig:teaser']}. These five groups have reflectional symmetry which forces the corresponding part of the boundary to remain a straight line, leading to a more-restricted domain and less elaborate tile shapes. The remaining four have a completely fixed boundary.
• Figure 2: Overview of our method. (a) the optimization pipeline: the parameters $\theta$ control our novel differentiable tile representation (Section \ref{['ss:diff_tile']}), which outputs the vertices $\mathbf{V}^\theta$ of the mesh, guaranteed to represent a valid tile, with no self-overlaps, that can tile the plane perfectly. The mesh is textured using the texture image $\mathcal{I}$ and rendered via a differentiable renderer, producing the final render $\mathcal{R}$ of the textured tile. The render is fed into Score Distillation Sampling dreamfusion along with the text prompt. From there, gradients are back-propagated in reverse to the optimizable parameters - $\theta$ and $\mathcal{I}$. (b) production of the final image of the tiling: After optimization is finished, we take copies of the resulting tile, color them with different colors, and tile the plane with them, to produce the final images shown in this paper.
• Figure 3: Fabrication. The output of our method can be readily used to 3D print physical tiles.
• Figure 4: The way in which copies of the tile connect to one another defines correspondences between different parts of the boundary, and as a result boundary conditions.
• Figure 5: Different boundary conditions for the same symmetry group. Different choices of the boundary conditions, Equation \ref{['eq:boundary_constraints']}, lead to different families of tiles for the same wallpaper group, each family having different correspondences on its boundary (visualized with matching colors): left has two parts of its boundary in correspondence to one another, right has three parts.

Theorems & Definitions (2)

• Theorem 1
• proof