COIN++: Neural Compression Across Modalities

TL;DR

COIN++ introduces a cross-modality neural compression framework that represents data as coordinates-to-features implicit neural representations encoded by a shared base network whose instance-specific information is stored as lightweight modulations. Through meta-learning, patch-based training, and efficient quantization/entropy coding of modulations, the approach achieves significant speedups and compression gains over prior INR-based methods and several traditional codecs, while remaining applicable to images, audio, medical, and climate data. Although it does not yet surpass state-of-the-art codecs in all modalities, COIN++ demonstrates strong potential for flexible neural codecs across diverse domains and highlights clear directions for improving entropy modeling and scalability. The work emphasizes the value of shared structure via a base network and per-instance modulations for efficient cross-modal compression.

Abstract

Neural compression algorithms are typically based on autoencoders that require specialized encoder and decoder architectures for different data modalities. In this paper, we propose COIN++, a neural compression framework that seamlessly handles a wide range of data modalities. Our approach is based on converting data to implicit neural representations, i.e. neural functions that map coordinates (such as pixel locations) to features (such as RGB values). Then, instead of storing the weights of the implicit neural representation directly, we store modulations applied to a meta-learned base network as a compressed code for the data. We further quantize and entropy code these modulations, leading to large compression gains while reducing encoding time by two orders of magnitude compared to baselines. We empirically demonstrate the feasibility of our method by compressing various data modalities, from images and audio to medical and climate data.

Figures (18)

• Figure 1: COIN++ converts a wide range of data modalities to neural networks via optimization and then stores the parameters of these neural networks as compressed codes for the data. Different data modalities can be compressed by simply changing the input and output dimensions of the neural networks.
• Figure 2: COIN++ architecture. Latent modulations $\phi$ (in green) are mapped through a hypernetwork to modulations (in blue) which are added to activations of the base network $f_\theta$ (in white) to parameterize a single function that can be evaluated at coordinates $\mathbf{x}$ to obtain features $\mathbf{y}$.
• Figure 3: By applying modulations $\phi^{(1)}$, $\phi^{(2)}$, $\phi^{(3)}$ to a base network $f_\theta$, we obtain different functions that can be decoded into datapoints $\mathbf{d}^{(1)}$, $\mathbf{d}^{(2)}$, $\mathbf{d}^{(3)}$ by evaluating the functions at various coordinates. While we show images in this figure, the same principle can be applied to a range of data modalities.
• Figure 4: (Left) Starting from a random initialization $\theta$, we meta-learn parameters $\theta^*$ of the base network (with training progress shown as a solid line) such that modulations $\phi$ can easily be fit in a few gradient steps (with fitting shown in dashed lines). (Right) During training we sample patches randomly, while at test time we partition the datapoint into patches and fit modulations to each patch.
• Figure 5: (Left) Test PSNR on CIFAR10 during training using a fixed number of modulations (see appendix for experimental details). Our method outperforms both baselines, improving PSNR by 2dB for the same number of parameters. (Right) Comparison of using shifts, scales and scales & shifts for modulations on MNIST (note that shifts and scales & shifts overlap). As can be seen, shifts perform significantly better than scales.