Learned Compression of Point Cloud Geometry and Attributes in a Single Model through Multimodal Rate-Control

TL;DR

The paper tackles efficient joint compression of point cloud geometry and attributes by introducing a single adaptive autoencoder that embeds both modalities into a unified latent space and uses per-point quality maps for local rate-distortion control. It extends transform coding with a multimodal conditioning framework, defining a rate-distortion objective $\mathcal{L}$ and a sparse architecture with conditional feature extraction and upsampling/pruning to maintain sparsity. The approach achieves competitive geometric and attribute quality compared to state-of-the-art learned methods while significantly reducing encoding latency, and enables view-dependent or region-of-interest encoding through per-point conditioning. This work highlights the practical impact of unified, variable-rate, multimodal point cloud compression for interactive streaming and immersive applications, and points to future directions in per-point importance and task-driven conditioning.

Abstract

Point cloud compression is essential to experience volumetric multimedia as it drastically reduces the required streaming data rates. Point attributes, specifically colors, extend the challenge of lossy compression beyond geometric representation to achieving joint reconstruction of texture and geometry. State-of-the-art methods separate geometry and attributes to compress them individually. This comes at a computational cost, requiring an encoder and a decoder for each modality. Additionally, as attribute compression methods require the same geometry for encoding and decoding, the encoder emulates the decoder-side geometry reconstruction as an input step to project and compress the attributes. In this work, we propose to learn joint compression of geometry and attributes using a single, adaptive autoencoder model, embedding both modalities into a unified latent space which is then entropy encoded. Key to the technique is to replace the search for trade-offs between rate, attribute quality and geometry quality, through conditioning the model on the desired qualities of both modalities, bypassing the need for training model ensembles. To differentiate important point cloud regions during encoding or to allow view-dependent compression for user-centered streaming, conditioning is pointwise, which allows for local quality and rate variation. Our evaluation shows comparable performance to state-of-the-art compression methods for geometry and attributes, while reducing complexity compared to related compression methods.

Figures (13)

• Figure 1: Variable-Rate Coding using a joint architecture for geometry and attributes, allows to freely select attribute and geometric quality with only a single model. The figure shows one mapping (path) of attribute and geometry quality configurations $q^{(G/A)}$ that compose a quality (PCQM) Pareto front with the corresponding rates given in bit per point (bpp).
• Figure 2: Operational diagrams of the Mean-Scale Hyperprior model balle2018variationalminnen2018joint and the conditional addition presented by song2021variable.
• Figure 3: Model Architecture, materializing the operating diagram in Fig. \ref{['fig:Song']}. For Convolutions, $b^3$ indicates filters with kernel size $b$ and dimension $3$. The arrows $\uparrow/\downarrow$ symbolize upsampling and downsampling, respectively. A more detailed version of the architecture containing implementation details is given in the supplementary materials.
• Figure 4: Conditioned feature extraction (CFE) block. A condition map $\mathbf{c}^{(k)} = [\alpha^{(k)}, \beta^{(k)}]$ is derived from the quality map $\mathbf{q}^{(k)}$ at scale $k$ for element-wise scaling and shifting of features according to \ref{['eq:CFE']}. Consequently, the module learns a channel-wise weighting of features in $\mathbf{t}^{(k)}$ according to the specified qualities $\mathbf{q}^{(A)}$ and $\mathbf{q}^{(G)}$.
• Figure 5: The Upsampling and Pruning (UP) block generates a locally dense geometry $\mathbf{\hat{d}}^{(k)}$ from the lower scale points $\mathbf{\hat{g}}^{(k+1)}$. Per-point occupancy probabilities $\mathbf{p}^{(k)}$ are predicted from the upsampled and transformed features $\mathbf{f}^{(k)}$. Finally, the top-k most probable points are selected to prune the dense point cloud $(\mathbf{d}^{(k)}, \mathbf{f}^{(k)})$, arriving at $(\mathbf{\hat{g}}^{(k)}, \mathbf{\hat{f}}^{(k)})$.