Ninja Codes: Neurally Generated Fiducial Markers for Stealthy 6-DoF Tracking

TL;DR

Ninja Codes address the need for inconspicuous $6$-DoF tracking by neurally generating fiducial markers that blend into real-world textures. The authors propose an end-to-end trainable pipeline (encoder, region/corner detectors, decoder, adversary) with differentiable noise modeling to simulate printing and imaging perturbations. They demonstrate centimeter-level localization and near $1^ ext{\circ}$ pose accuracy on indoor scenes, while revealing artifacts and privacy considerations that emerge when markers are made visually subtle. The work expands the applicability of precise location tracking to aesthetically sensitive settings and outlines extensions like reverse encoders, color calibration, and new materials to broaden deployment and resilience.

Abstract

In this paper we describe Ninja Codes, neurally-generated fiducial markers that can be made to naturally blend into various real-world environments. An encoder network converts arbitrary images into Ninja Codes by applying visually modest alterations; the resulting codes, printed and pasted onto surfaces, can provide stealthy 6-DoF location tracking for a wide range of applications including augmented reality, robotics, motion-based user interfaces, etc. Ninja Codes can be printed using off-the-shelf color printers on regular printing paper, and can be detected using any device equipped with a modern RGB camera and capable of running inference. Using an end-to-end process inspired by prior work on deep steganography, we jointly train a series of network modules that perform the creation and detection of Ninja Codes. Through experiments, we demonstrate Ninja Codes' ability to provide reliable location tracking under common indoor lighting conditions, while successfully concealing themselves within diverse environmental textures. We expect Ninja Codes to offer particular value in scenarios where the conspicuous appearances of conventional fiducial markers make them undesirable for aesthetic and other reasons.

Figures (11)

• Figure 1: Ninja Codes end-to-end training architecture. A total of five modules are trained simultaneously: encoder, decoder, region detector, corner detector, and adversary. A series of differentiable noise functions are applied during the training process, to simulate perturbations that arise when printing and capturing the codes in real-world situations.
• Figure 2: Noise functions to simulate perturbations. Perturbations owing to the printing method/material are simulated using color shift and synthetic specular reflection. Perturbations owing to camera capture are simulated using color shift, Gaussian blur, Gaussian noise, and JPEG compression.
• Figure 3: We employ a two-phase training process. After the first phase, the encoder will have learned to produce brightly-colored Ninja Codes (top left). After the second phase, the encoder will produce Ninja Codes closer to the cover image albeit with residual visual artifacts (bottom). Artifact saliency can be controlled by adjusting hyperparameters.
• Figure 4: The 25 digital images used to evaluate code detection performance, each with a single Ninja Code (here, generated using the $NC_{300}$ encoder) placed at the center (top). The eight camera positions (bottom).
• Figure 5: Poster boards used to evaluate 6-DoF tracking performance, with four Ninja Codes (generated using the $NC_{200}$ encoder) placed at the corners (top). The three camera positions (bottom).