Variable Aperture Bokeh Rendering via Customized Focal Plane Guidance

TL;DR

This paper has proposed an effective controllable bokeh rendering method, and contributed a Variable Aperture Bokeh Dataset (VABD), and demonstrated that the customized focal plane together aperture prompt can bootstrap model to simulate realistic bokeh effect.

Abstract

Bokeh rendering is one of the most popular techniques in photography. It can make photographs visually appealing, forcing users to focus their attentions on particular area of image. However, achieving satisfactory bokeh effect usually presents significant challenge, since mobile cameras with restricted optical systems are constrained, while expensive high-end DSLR lens with large aperture should be needed. Therefore, many deep learning-based computational photography methods have been developed to mimic the bokeh effect in recent years. Nevertheless, most of these methods were limited to rendering bokeh effect in certain single aperture. There lacks user-friendly bokeh rendering method that can provide precise focal plane control and customised bokeh generation. There as well lacks authentic realistic bokeh dataset that can potentially promote bokeh learning on variable apertures. To address these two issues, in this paper, we have proposed an effective controllable bokeh rendering method, and contributed a Variable Aperture Bokeh Dataset (VABD). In the proposed method, user can customize focal plane to accurately locate concerned subjects and select target aperture information for bokeh rendering. Experimental results on public EBB! benchmark dataset and our constructed dataset VABD have demonstrated that the customized focal plane together aperture prompt can bootstrap model to simulate realistic bokeh effect. The proposed method has achieved competitive state-of-the-art performance with only 4.4M parameters, which is much lighter than mainstream computational bokeh models. The contributed dataset and source codes will be released on github https://github.com/MoTong-AI-studio/VABM.

Figures (9)

• Figure 1: The effect of different aperture sizes on bokeh effects. (a) Narrow aperture "f/8". (b) Aperture "f/2.8". (c) Large aperture "f/1.8". The larger the aperture, the more pronounced bokeh effect.
• Figure 2: Illustration on physical imaging principles. Objects within certain depth ranges around focal plane can be clearly projected on image plane. Objects outside the focal plane will be out of focus, forming the observed confusion circle.
• Figure 3: The framework of the proposed method. Depth map and focal plane are generated in DFMG, which are then fed into the Variable Aperture Bokeh Model (VABM) for fusion, providing the model with information such as depth relationship and focused subject to guide the bokeh rendering process. And the target lens information is fully fused with the features in each LFMB to achieve the effect of selectable target lens.
• Figure 4: Structures of Multiple information fusion block (MIFB) and Lens Fusion Block (LFB). MIFB extracts and fully fuses the global information of the depth map and the local information of the focal plane to provide guidance for subsequent processes. LFB embeds the lens information into the model through sinusoidal embedding to distinguish different target lenses.
• Figure 5: Example on the effectiveness of MIFB. the "Before MIFB" is the feature map obtained without fusing the depth of field map and focal plane, while the "After MIFB" is the feature map obtained after fusion.Compared with Before MIFB, the attention of the sky background with similar depth of field in After MIFB is closer, and more attention is paid to the location of the subject.