Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

EchoMark: Perceptual Acoustic Environment Transfer with Watermark-Embedded Room Impulse Response

Chenpei Huang, Lingfeng Yao, Kyu In Lee, Lan Emily Zhang, Xun Chen, Miao Pan

TL;DR

EchoMark addresses the risk of perceptual environment transfer by embedding a watermark directly in the latent RIR used for AEM. It jointly optimizes RIR reconstruction and watermark recovery via a Conformer-based RIR encoder, a HiFiGAN/DECOR-inspired generator, and a separate watermark detector, achieving perceptual fidelity on par with state-of-the-art RIR estimators while ensuring robust watermark detection (≥99%) and low BER (≤0.3%). Human listening tests yield a MOS of 4.22, confirming strong perceptual alignment between EchoMark and target environments. The method supports sequential watermarking to encode longer messages and demonstrates robustness to noise, room type, and speaker variability, highlighting practical applicability and safeguards for AEM services.

Abstract

Acoustic Environment Matching (AEM) is the task of transferring clean audio into a target acoustic environment, enabling engaging applications such as audio dubbing and auditory immersive virtual reality (VR). Recovering similar room impulse response (RIR) directly from reverberant speech offers more accessible and flexible AEM solution. However, this capability also introduces vulnerabilities of arbitrary ``relocation" if misused by malicious user, such as facilitating advanced voice spoofing attacks or undermining the authenticity of recorded evidence. To address this issue, we propose EchoMark, the first deep learning-based AEM framework that generates perceptually similar RIRs with embedded watermark. Our design tackle the challenges posed by variable RIR characteristics, such as different durations and energy decays, by operating in the latent domain. By jointly optimizing the model with a perceptual loss for RIR reconstruction and a loss for watermark detection, EchoMark achieves both high-quality environment transfer and reliable watermark recovery. Experiments on diverse datasets validate that EchoMark achieves room acoustic parameter matching performance comparable to FiNS, the state-of-the-art RIR estimator. Furthermore, a high Mean Opinion Score (MOS) of 4.22 out of 5, watermark detection accuracy exceeding 99\%, and bit error rates (BER) below 0.3\% collectively demonstrate the effectiveness of EchoMark in preserving perceptual quality while ensuring reliable watermark embedding.

EchoMark: Perceptual Acoustic Environment Transfer with Watermark-Embedded Room Impulse Response

TL;DR

EchoMark addresses the risk of perceptual environment transfer by embedding a watermark directly in the latent RIR used for AEM. It jointly optimizes RIR reconstruction and watermark recovery via a Conformer-based RIR encoder, a HiFiGAN/DECOR-inspired generator, and a separate watermark detector, achieving perceptual fidelity on par with state-of-the-art RIR estimators while ensuring robust watermark detection (≥99%) and low BER (≤0.3%). Human listening tests yield a MOS of 4.22, confirming strong perceptual alignment between EchoMark and target environments. The method supports sequential watermarking to encode longer messages and demonstrates robustness to noise, room type, and speaker variability, highlighting practical applicability and safeguards for AEM services.

Abstract

Acoustic Environment Matching (AEM) is the task of transferring clean audio into a target acoustic environment, enabling engaging applications such as audio dubbing and auditory immersive virtual reality (VR). Recovering similar room impulse response (RIR) directly from reverberant speech offers more accessible and flexible AEM solution. However, this capability also introduces vulnerabilities of arbitrary ``relocation" if misused by malicious user, such as facilitating advanced voice spoofing attacks or undermining the authenticity of recorded evidence. To address this issue, we propose EchoMark, the first deep learning-based AEM framework that generates perceptually similar RIRs with embedded watermark. Our design tackle the challenges posed by variable RIR characteristics, such as different durations and energy decays, by operating in the latent domain. By jointly optimizing the model with a perceptual loss for RIR reconstruction and a loss for watermark detection, EchoMark achieves both high-quality environment transfer and reliable watermark recovery. Experiments on diverse datasets validate that EchoMark achieves room acoustic parameter matching performance comparable to FiNS, the state-of-the-art RIR estimator. Furthermore, a high Mean Opinion Score (MOS) of 4.22 out of 5, watermark detection accuracy exceeding 99\%, and bit error rates (BER) below 0.3\% collectively demonstrate the effectiveness of EchoMark in preserving perceptual quality while ensuring reliable watermark embedding.

Paper Structure

This paper contains 41 sections, 13 equations, 10 figures, 1 table.

Table of Contents

  1. Introduction
  2. Background
  3. Acoustic Environment Matching.
  4. Blind and Informed RIR Estimation.
  5. Audio Watermarking.
  6. System Overview
  7. Problem Formulation
  8. Natural Reverberant Speech Generation.
  9. Target RIR Generation with Watermark.
  10. Watermark Detection on AEM Speech.
  11. Sequential Mode for Long Message.
  12. EchoMark
  13. RIR Encoder.
  14. RIR Generator.
  15. Watermark Detector.
  16. ...and 26 more sections

Figures (10)

  • Figure 1: EchoMark system overview in perceptual environment transfer using watermarked RIR.
  • Figure 2: EchoMark's alternative sequential mode.
  • Figure 3: EchoMark RIR Encoder. FFN: Feed-Forward Network; MHSA: Multi-Head Self-Attention; CONV: Convolutional Layer; LN: Layer Normalization.
  • Figure 4: EchoMark RIR Generator. Early: HiFiGAN decoder architecture; Late: Decor architecture.
  • Figure 5: EchoMark Watermark Detector. Encoder architecture reused in dashed box.
  • ...and 5 more figures