Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

TimberAgent: Gram-Guided Retrieval for Executable Music Effect Control

Shihao He, Yihan Xia, Fang Liu, Taotao Wang, Shengli Zhang

TL;DR

This work interprets results as benchmark evidence that texture-aware retrieval is useful for editable audio effect control, while broader personalization and real-audio robustness claims remain outside the verified evidence presented here.

Abstract

Digital audio workstations expose rich effect chains, yet a semantic gap remains between perceptual user intent and low-level signal-processing parameters. We study retrieval-grounded audio effect control, where the output is an editable plugin configuration rather than a finalized waveform. Our focus is Texture Resonance Retrieval (TRR), an audio representation built from Gram matrices of projected mid-level Wav2Vec2 activations. This design preserves texture-relevant co-activation structure. We evaluate TRR on a guitar-effects benchmark with 1,063 candidate presets and 204 queries. The evaluation follows Protocol-A, a cross-validation scheme that prevents train-test leakage. We compare TRR against CLAP and internal retrieval baselines (Wav2Vec-RAG, Text-RAG, FeatureNN-RAG), using min-max normalized metrics grounded in physical DSP parameter ranges. Ablation studies validate TRR's core design choices: projection dimensionality, layer selection, and projection type. A near-duplicate sensitivity analysis confirms that results are robust to trivial knowledge-base matches. TRR achieves the lowest normalized parameter error among evaluated methods. A multiple-stimulus listening study with 26 participants provides complementary perceptual evidence. We interpret these results as benchmark evidence that texture-aware retrieval is useful for editable audio effect control, while broader personalization and real-audio robustness claims remain outside the verified evidence presented here.

TimberAgent: Gram-Guided Retrieval for Executable Music Effect Control

TL;DR

This work interprets results as benchmark evidence that texture-aware retrieval is useful for editable audio effect control, while broader personalization and real-audio robustness claims remain outside the verified evidence presented here.

Abstract

Digital audio workstations expose rich effect chains, yet a semantic gap remains between perceptual user intent and low-level signal-processing parameters. We study retrieval-grounded audio effect control, where the output is an editable plugin configuration rather than a finalized waveform. Our focus is Texture Resonance Retrieval (TRR), an audio representation built from Gram matrices of projected mid-level Wav2Vec2 activations. This design preserves texture-relevant co-activation structure. We evaluate TRR on a guitar-effects benchmark with 1,063 candidate presets and 204 queries. The evaluation follows Protocol-A, a cross-validation scheme that prevents train-test leakage. We compare TRR against CLAP and internal retrieval baselines (Wav2Vec-RAG, Text-RAG, FeatureNN-RAG), using min-max normalized metrics grounded in physical DSP parameter ranges. Ablation studies validate TRR's core design choices: projection dimensionality, layer selection, and projection type. A near-duplicate sensitivity analysis confirms that results are robust to trivial knowledge-base matches. TRR achieves the lowest normalized parameter error among evaluated methods. A multiple-stimulus listening study with 26 participants provides complementary perceptual evidence. We interpret these results as benchmark evidence that texture-aware retrieval is useful for editable audio effect control, while broader personalization and real-audio robustness claims remain outside the verified evidence presented here.
Paper Structure (52 sections, 9 equations, 5 figures, 6 tables, 1 algorithm)

This paper contains 52 sections, 9 equations, 5 figures, 6 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Perceptual similarity vs. parameter transferability.
  4. Problem Formulation
  5. Signal Processing Dynamics
  6. Goal and Constraints
  7. Operational objective (without probabilistic semantics).
  8. The Retrieval-Based Approach
  9. Design Principle 1 (Well-defined synthesis under missing modalities).
  10. Methodology
  11. System Architecture
  12. Design Principle 2 (Runtime isolation).
  13. Texture Resonance Retrieval (TRR)
  14. Properties and limitations.
  15. Property 3 (Alignment-insensitive texture statistics).
  16. ...and 37 more sections

Figures (5)

  • Figure 1: System overview of the retrieval-grounded audio effect control pipeline. The real-time DSP engine uses a six-module serial effect chain (as shown in the framework), where each module can be independently bypassed. Retrieved parameters are validated against per-module physical bounds (Eq. \ref{['eq:feasible_set']}) before being applied.
  • Figure 2: Gram matrix heatmaps for representative query presets. Each matrix shows co-activation structure across projected channels, revealing distinct texture patterns for different effect styles.
  • Figure 3: Trial 1: condition-level score distributions (boxplot) for style matching.
  • Figure 4: Trials 6--10: condition-level score distributions.
  • Figure 5: t-SNE comparison of TRR (Gram matrix) and Wav2Vec2 mean-pooled embeddings on the full dataset, colored by style label. TRR exhibits tighter within-style clustering, supporting its utility as a texture-aware retrieval prior.