Rasterized Steered Mixture of Experts for Efficient 2D Image Regression

TL;DR

Rasterized SMoE (R-SMoE) combines the edge-aware, sparse gating of Steered Mixture of Experts with a tile-based rasterization strategy to accelerate 2D image regression, denoising, and native super-resolution. By restricting kernel influence to block-relevant subsets and precomputing kernel coverage, it achieves orders-of-magnitude speedups and memory savings while maintaining reconstruction fidelity comparable to or better than global SMoE and Gaussian Splatting baselines. The approach includes segmentation-guided initialization and multi-model fusion to enhance denoising, yielding measurable gains in PSNR/SSIM with reduced artifacts. These advancements offer a practical, scalable solution for high-quality image regression tasks and potentially extendable to higher-dimensional data, albeit with some limitations on very high-frequency textures and encoding time.

Abstract

The Steered Mixture of Experts regression framework has demonstrated strong performance in image reconstruction, compression, denoising, and super-resolution. However, its high computational cost limits practical applications. This work introduces a rasterization-based optimization strategy that combines the efficiency of rasterized Gaussian kernel rendering with the edge-aware gating mechanism of the Steered Mixture of Experts. The proposed method is designed to accelerate two-dimensional image regression while maintaining the model's inherent sparsity and reconstruction quality. By replacing global iterative optimization with a rasterized formulation, the method achieves significantly faster parameter updates and more memory-efficient model representations. In addition, the proposed framework supports applications such as native super-resolution and image denoising, which are not directly achievable with standard rasterized Gaussian kernel approaches. The combination of fast rasterized optimization with the edge-aware structure of the Steered Mixture of Experts provides a new balance between computational efficiency and reconstruction fidelity for two-dimensional image processing tasks.

Figures (14)

• Figure 1: Illustration of the edge-aware Steered Mixture of Experts (SMoE) model applied to image compression and denoising at 0.43 bits per pixel (bpp). Figure adapted fromfleig_edge-aware_2022.
• Figure 2: Comparison of kernel behaviors and reconstruction results for the RBF and SMoE models demonstrated on denoising and super resolution tasks.
• Figure 3: (a) Left: 2D Gaussian initialization with five colored circles representing Gaussian kernels. Middle: The bounding boxes indicate the coverage of blocks affected by the corresponding Gaussian kernels. Right: The block $b_n$ is reconstructed by the corresponding affected kernel set $\mathcal{K}_n$. Kernels are represented by distinct colors and shapes. (b) Gaussian kernels are represented as ellipses with varying axes. The coverage of affected blocks is shown by square boxes aligned with the centers of the kernels (ellipses), where the box side length matches the long axis of the corresponding kernel (ellipse).
• Figure 4: PSNR versus average number of kernels per block. The x-axis shows the average number of kernels used to render each block, highlighting the efficiency of selecting a subset rather than employing all available kernels. The figure offers a comprehensive comparison across four models: R-SMoE, G-SMoE, GaussianImage, and RBF, demonstrating the substantial reduction in computational demand achieved by R-SMoE. A red rectangle emphasizes a detailed comparison between R-SMoE and GaussianImage, illustrating subtle yet crucial differences in FLOPs. Our method delivers up to a 6 dB PSNR improvement over state-of-the-art methods when operating under similar average kernel counts, underscoring both quality and efficiency gains.
• Figure 5: Performance scaling with kernel pool size on PSNR, SSIM, and LPIPS. The curves show how image quality improves with larger kernel pools while highlighting R-SMoE’s ability to maintain high visual fidelity with fewer kernels, demonstrating its efficiency–quality trade-off advantage.