ArCSEM: Artistic Colorization of SEM Images via Gaussian Splatting

TL;DR

This work explores several scene representation techniques and achieves high-quality colorized novel view synthesis of a SEM scene, and enables an artist to color a single or few views of a sequence and automatically retrieve a fully colored scene or video.

Abstract

Scanning Electron Microscopes (SEMs) are widely renowned for their ability to analyze the surface structures of microscopic objects, offering the capability to capture highly detailed, yet only grayscale, images. To create more expressive and realistic illustrations, these images are typically manually colorized by an artist with the support of image editing software. This task becomes highly laborious when multiple images of a scanned object require colorization. We propose facilitating this process by using the underlying 3D structure of the microscopic scene to propagate the color information to all the captured images, from as little as one colorized view. We explore several scene representation techniques and achieve high-quality colorized novel view synthesis of a SEM scene. In contrast to prior work, there is no manual intervention or labelling involved in obtaining the 3D representation. This enables an artist to color a single or few views of a sequence and automatically retrieve a fully colored scene or video. Project page: https://ronly2460.github.io/ArCSEM

Figures (9)

• Figure 1: Our dataset. (a) A subset of 18 out of 32 grayscale images, arranged left to right in the first two rows, and front to top in the bottom row. (b) All manually colored images shown in the following order: leftmost, center, rightmost, angled, and top view
• Figure 2: Overview of our two-stage approach: (a) Grayscale training: We fit 2DGS huang20242d with an image-specific affine color transformation to the grayscale images calibrated with RealityCapture realitycapture. (b) Colorization: 2DGS depth maps are used to project colors from limited manually colorized images into 3D space as pseudo-colors. Together with the input color views, the pseudo-colors guide the colorization of the grayscale model via L1, TCM, and CCM loss functions.
• Figure 3: Grayscale novel views. The first two rows show novel views in the lateral trajectory and the bottom row indicates the view from the top. All models are trained at $3072 \!\times\! 2048$. Red squares highlight areas with illumination differences. ACT in Ours effectively normalizes the differences across views.
• Figure 4: Grayscale novel views with closeups. The left side displays the nearest Ground Truth (GT) image along with its closeups. This novel view is precisely between two adjacent GTs. The right side shows the generated novel views and their corresponding closeups. Our method (2DGS+ACT) exhibits superior quality in synthesising novel views.
• Figure 5: Predicted depth maps of novel views. The view in the first row is sampled from the horizontal trajectory, while the view in the second is sampled from the vertical trajectory. All other methods failed to predict depths in the vertical trajectory. The depth map accuracy is essential for correctly projecting colors into the 3D space.