Analysis of Coding Gain Due to In-Loop Reshaping

TL;DR

This work provides the first theoretical justification for coding gains from in-loop reshaping in hybrid video codecs. By modeling a simplified one-piece range-expansion reshaper, the authors derive a closed-form RD relation that shows a distortion reduction of $\mathrm{MSE} \approx (q/k)^2/12$ and an entropy increase of $H^{(1)} \approx H^{(0)}+\log_2 k$. Crucially, they show that gains arise only when the entropy coder is suboptimal, with a PSNR gain given by $\Delta \mathrm{PSNR} = 20(1-\eta)\log_{10} k$, where $\eta$ captures the coder’s suboptimality; experiments with a simplified codec and standard test sequences verify the predicted gains. The results illuminate why in-loop reshaping improves coding efficiency in practice and offer guidance for real codecs (e.g., VVC LMCS) by highlighting the role of entropy coding; they also discuss extensions to multi-piece reshapers and practical considerations such as quantization and evaluation metrics.

Abstract

Reshaping, a point operation that alters the characteristics of signals, has been shown capable of improving the compression ratio in video coding practices. Out-of-loop reshaping that directly modifies the input video signal was first adopted as the supplemental enhancement information (SEI) for the HEVC/H.265 without the need to alter the core design of the video codec. VVC/H.266 further improves the coding efficiency by adopting in-loop reshaping that modifies the residual signal being processed in the hybrid coding loop. In this paper, we theoretically analyze the rate-distortion performance of the in-loop reshaping and use experiments to verify the theoretical result. We prove that the in-loop reshaping can improve coding efficiency when the entropy coder adopted in the coding pipeline is suboptimal, which is in line with the practical scenarios that video codecs operate in. We derive the PSNR gain in a closed form and show that the theoretically predicted gain is consistent with that measured from experiments using standard testing video sequences.

Figures (7)

• Figure 1: Pipelines for video encoders with (a) out-of-loop reshapers and (b) in-loop reshapers. (c) Simplified schematic for the in-loop reshaper with blocks and symbols that are needed for the rate--distortion analysis.
• Figure 2: A typical one-piece (a) forward reshaping function $g(x) \in [0, M]$, where $M = 2^n-1$, and (b) its corresponding backward reshaping function $g^{-1}(x) \in [aM, bM]$, where $0 < a < b < 1$.
• Figure 3: (a) MSE--entropy (H--D) curves overlap before and after reshaping, indicating that reshaping does not lead to coding gain for optimal entropy coders. (b) MSE--bitrate (R--D) curves before and after reshaping with a range expansion factor $k=1.5$ for a suboptimal entropy coder with a slope ratio $\eta=0.83$. The gap indicates that reshaping can lead to coding gain for a suboptimal entropy coder.
• Figure 4: Operating points $(H^{(0)}, R^{(0)})$ and $(H^{(1)}, R^{(1)})$ on bitrate function $u(\cdot)$ (not shown for clarity) before and after reshaping on the bitrate--entropy plane, respectively. The secant from the origin to $(H^{(0)}, R^{(0)})$ has a slope of $m_0 > 1$ and the secant from $(H^{(0)}, R^{(0)})$ to $(H^{(1)}, R^{(1)})$ has a slope of $1 \leq m_{0,1} < m_1$. The very fact that the slope ratio $\eta = m_{0,1} / m_0 < 1$ ensures that reshaping can always lead to coding gain when suboptimal entropy coders are used.
• Figure 5: Example plots from sequence BasketballDrill for measuring (a)--(b) the empirical PSNR gain and (c) the quantities needed for predicting the theoretical PSNR gain. (a) Rate--distortion (R--D) curves before and after reshaping. Each circle corresponds to a specific quantization step. (b) Zoomed-in version of (a). The plots reveal that reshaping causes the R--D curve to move to the top-right corner of the R--D plane, leading to an increased measured/empirical PSNR as predicted by our theoretical result \ref{['eq:psnr_gain_theory']}. (c) Corresponding bitrate--entropy (R--H) curve, which is contributed by the R--H points before (blue dashed) and after (red dashed) reshaping. The R--H curve sits above the diagonal line, confirming the suboptimality of the entropy coder stated in \ref{['eq:suboptimal_mapping']}. The two operating points $(\hat{H}^{(0)}, \hat{R}^{(0)})$ ("$\circ$") and $(\hat{H}^{(1)}, \hat{R}^{(1)})$ ("$\bullet$") for predicting the theoretical PSNR gain are annotated on the R--H curve.