Audio Inpainting in Time-Frequency Domain with Phase-Aware Prior

TL;DR

This work tackles spectrogram inpainting in the time-frequency domain by introducing U-PHAIN-TF, a phase-aware prior that leverages instantaneous frequency and the iPCTV penalty to preserve sinusoidal TF components. The method is formulated as a convex optimization solved with the generalized Chambolle--Pock algorithm, combining phase correction, time-variational regularization, and IF updates. It demonstrates superior objective metrics (SNR, ODG) and MOS-like listening-test performance compared with DPAI and Janssen-TF, while achieving substantially lower computational load. The approach offers a practical and efficient solution for reconstructing missing TF-domain data in audio applications and codecs.

Abstract

The so-called audio inpainting problem in the time domain refers to estimating missing segments of samples within a signal. Over the years, several methods have been developed for such type of audio inpainting. In contrast to this case, a time-frequency variant of inpainting appeared in the literature, where the challenge is to reconstruct missing spectrogram columns with reliable information. We propose a method to address this time-frequency audio inpainting problem. Our approach is based on the recently introduced phase-aware signal prior that exploits an estimate of the instantaneous frequency. An optimization problem is formulated and solved using the generalized Chambolle-Pock algorithm. The proposed method is evaluated both objectively and subjectively against other time-frequency inpainting methods, specifically a deep-prior neural network and the autoregression-based approach known as Janssen-TF. Our proposed approach surpassed these methods in the objective evaluation as well as in the conducted listening test. Moreover, this outcome is achieved with a substantially reduced computational requirement compared to alternative methods.

Figures (12)

• Figure 1: Audio inpainting problem in the time domain (top) and in the time-frequency domain (bottom).
• Figure 2: Different masks used in experiments. Each mask has five gaps with length ranging from 1 missing column to 6 missing columns. Here, only the second mask (left), fourth mask (middle), and sixth mask (right) is shown.
• Figure 3: Objective results for U-PHAIN-TF with different setting of lambda. The ODG results are computed on the DPAI dataset with all masks applied to each example. For each setting of $\lambda$, the results are averaged across all examples and masks.
• Figure 4: Results from U-PHAIN-TF with different settings of inner iterations, while the number of outer iterations was fixed to $J=10$. The SNR (left) and ODG (right) are computed on the DPAI dataset and averaged across all examples and masks.
• Figure 5: Results from U-PHAIN-TF with different settings of outer iterations, while the number of inner iterations was fixed to $I=500$. The SNR (left) and ODG (right) are computed on the DPAI dataset and averaged across all examples and masks.