D'OH: Decoder-Only Random Hypernetworks for Implicit Neural Representations

TL;DR

This paper proposes to use a novel runtime decoder-only hypernetwork - that uses no offline training data - to better exploit cross-layer parameter redundancy and presents a strategy for the optimization of runtime deep implicit functions through a Decoder-Only randomly projected Hypernetwork (D'OH).

Abstract

Deep implicit functions have been found to be an effective tool for efficiently encoding all manner of natural signals. Their attractiveness stems from their ability to compactly represent signals with little to no offline training data. Instead, they leverage the implicit bias of deep networks to decouple hidden redundancies within the signal. In this paper, we explore the hypothesis that additional compression can be achieved by leveraging redundancies that exist between layers. We propose to use a novel runtime decoder-only hypernetwork - that uses no offline training data - to better exploit cross-layer parameter redundancy. Previous applications of hypernetworks with deep implicit functions have employed feed-forward encoder/decoder frameworks that rely on large offline datasets that do not generalize beyond the signals they were trained on. We instead present a strategy for the optimization of runtime deep implicit functions for single-instance signals through a Decoder-Only randomly projected Hypernetwork (D'OH). By directly changing the latent code dimension, we provide a natural way to vary the memory footprint of neural representations without the costly need for neural architecture search on a space of alternative low-rate structures.

Figures (17)

• Figure 1: Illustration of the proposed Decoder-Only Hypernetwork, which optimizes a low-dimensional latent code $z$ to generate weights for the target implicit neural representation (INR). This signal-agnostic framework operates without offline training data, relying solely on the target architecture and the specific data instance. Random projections act as the network decoder $\theta_d$, facilitating a compact code representation.
• Figure 1: Numerical comparison of layer variances between SIREN and the weights generated by D'OH (latent dim: 2000 and $\omega=30$). Our initialization closely matches the initialization of SIREN sitzmann_implicit_2020.
• Figure 2: Conventional Encoder-Decoder Hypernetworks are optimized offline on a signal class. The hypernetwork [$\theta_e, \theta_d$] is frozen and runtime predicts INR weights for new data instances, limiting generality for out-of-distribution signals. In contrast, a Decoder-Only Hypernetwork (see: Figure \ref{['fig:decoder-banner']}) is runtime optimized using only the target instance.
• Figure 2: Comparison of bits-per-pixel (BPP) for estimated memory footprint (parameters $\times$ bits-per-weight) [dotted] and memory after applying BZIP2 [solid] to a Python pickle of the quantized model. Rate-distortions generated by varying quantization level. The estimated is a close proxy to an actual entropy coder, but shows some discrepancy at low-rate and low-quantization levels where file overhead represent a larger proportion of code size. To account for this we report the estimated memory footprint for both D'OH and MLPs, which can be seen as a overhead-free limit for performance.
• Figure 3: Left: COIN architectures show variability in outcome necessitating a costly Neural Architecture Search to achieve maximal performance as in dupont_coin_2021. Right: Architectures that satisfy a bits-per-pixel constraint for COIN (Kodak index 2) using 16-bit quantization. There is a combinatorial increase in the search space for optimal architectures when considering different quantization levels (e.g. for quantization aware training (QAT) as in strumpler_implicit_2022gordon_quantizing_2023damodaran_rqat-inr_2023 as QAT requires fixed quantization during training rastegari_xnor-net_2016).