The Fibonacci Network: A Simple Alternative for Positional Encoding

TL;DR

It is shown that very simple MLPs can quite easily output a frequency when given input of the half-frequency and quarter-frequency, and a smarter construction of the network architecture together with a smart training method is sufficient to achieve similar results.

Abstract

Coordinate-based Multi-Layer Perceptrons (MLPs) are known to have difficulty reconstructing high frequencies of the training data. A common solution to this problem is Positional Encoding (PE), which has become quite popular. However, PE has drawbacks. It has high-frequency artifacts and adds another hyper-hyperparameter, just like batch normalization and dropout do. We believe that under certain circumstances PE is not necessary, and a smarter construction of the network architecture together with a smart training method is sufficient to achieve similar results. In this paper, we show that very simple MLPs can quite easily output a frequency when given input of the half-frequency and quarter-frequency. Using this, we design a network architecture in blocks, where the input to each block is the output of the two previous blocks along with the original input. We call this a {\it Fibonacci Network}. By training each block on the corresponding frequencies of the signal, we show that Fibonacci Networks can reconstruct arbitrarily high frequencies.

Figures (7)

• Figure 1: The MLP's reconstruction of sin(512x) (orange), compared with the ground truth (blue).
• Figure 2: The MLP's reconstruction with both x and sin(256x) as input.
• Figure 3: The MLP's reconstruction with x, sin(256x), and sin(128x) as input.
• Figure 4: Taking the network from hertz2021sape and training for more epochs. The training data are the red dots, the desired reconstructed signal is in blue, and our reconstruction is in orange.
• Figure 5: Output of hidden neurons as function of input to network $(x,\sin(256x),\sin(128x))$. Each plot is a separate neuron output.