3D View Optimization for Improving Image Aesthetics

TL;DR

This work tackles the challenge of improving image aesthetics by moving beyond traditional 2D cropping to a 3D view optimization approach. It extrapolates the input image, reconstructs a 3D scene, and then optimizes camera parameters and output aspect ratios to maximize an aesthetics score provided by a pre-trained evaluator, using CMA-ES for robust global search. Key contributions include introducing a practical extrapolation-and-reconstruction pipeline, a mask-based regularization to suppress out-of-view artifacts, and a comprehensive set of experiments showing superiority over existing 2D cropping methods both quantitatively and via user study. The approach enables larger, more informative viewpoints and demonstrates a significant step toward aesthetic retrospective rephotography with potential impact on automated photography tools and imaging software.

Abstract

Achieving aesthetically pleasing photography necessitates attention to multiple factors, including composition and capture conditions, which pose challenges to novices. Prior research has explored the enhancement of photo aesthetics post-capture through 2D manipulation techniques; however, these approaches offer limited search space for aesthetics. We introduce a pioneering method that employs 3D operations to simulate the conditions at the moment of capture retrospectively. Our approach extrapolates the input image and then reconstructs the 3D scene from the extrapolated image, followed by an optimization to identify camera parameters and image aspect ratios that yield the best 3D view with enhanced aesthetics. Comparative qualitative and quantitative assessments reveal that our method surpasses traditional 2D editing techniques with superior aesthetics.

Figures (6)

• Figure 1: Overview of our method. The input image is first extrapolated in the pre-processing and then fed to a 3D reconstruction method Shih3DP20 to obtain a point cloud that represents the 3D scene. Our method finds optimal camera parameters that maximize the aesthetic score wei-cvpr2018 of the rendered image through an optimization loop.
• Figure 2: Failure cases. Blue arrow: unphotographed regions. Green arrow: unsatisfactory objects created by image extrapolation. Red arrow: gap in the point cloud.
• Figure 3: Ablation study. We compare three variants; optimized using (i) Adam (2nd column), (ii) CMA-ES (3rd column), and (iii) CMA-ES with image scaling (4th column). The aesthetic scores (the larger, the better) under images are calculated using VEN.
• Figure 4: Qualitative comparison of our method with existing 2D cropping methods.
• Figure 5: Qualitative comparison of our method with existing 2D cropping methods.