Intensity and Texture Correction of Omnidirectional Image Using Camera Images for Indirect Augmented Reality

TL;DR

This work tackles the mismatch between past omnidirectional imagery and current scenes in Indirect AR by calibrating a single pre-captured omnidirectional image using a user-c captured panorama. The authors propose a pipeline that generates a panoramic image from a user video, aligns it to the pre-captured image via AKAZE/RANSAC, and then performs per-region histogram-based intensity correction for non-sky areas and sky texture restoration through inpainting-based sky synthesis. The method avoids database construction by leveraging current user-captured imagery to translate the past scene toward the present conditions, guided by semantic segmentation. Experiments across three locations show improved realism in some cases but reveal sensitivity to segmentation accuracy and panorama overlap, highlighting areas for improvement and future work in semantic segmentation and broader IAR evaluation.

Abstract

Augmented reality (AR) using camera images in mobile devices is becoming popular for tourism promotion. However, obstructions such as tourists appearing in the camera images may cause the camera pose estimation error, resulting in CG misalignment and reduced visibility of the contents. To avoid this problem, Indirect AR (IAR), which does not use real-time camera images, has been proposed. In this method, an omnidirectional image is captured and virtual objects are synthesized on the image in advance. Users can experience AR by viewing a scene extracted from the synthesized omnidirectional image according to the device's sensor. This enables robustness and high visibility. However, if the weather conditions and season in the pre-captured 360 images differs from the current weather conditions and season when AR is experienced, the realism of the AR experience is reduced. To overcome the problem, we propose a method for correcting the intensity and texture of a past omnidirectional image using camera images from mobile devices. We first perform semantic segmentation. We then reproduce the current sky pattern by panoramic image composition and inpainting. For the other areas, we correct the intensity by histogram matching. In experiments, we show the effectiveness of the proposed method using various scenes.

Figures (39)

• Figure 1: Flow of the proposed method.
• Figure 2: Example images extracted from video.
• Figure 3: Panoramic image generated with equirectangular projection.
• Figure 4: Transformed panoramic image.
• Figure 5: Pre-captured omnidirectional image.