Audio Palette: A Diffusion Transformer with Multi-Signal Conditioning for Controllable Foley Synthesis
Junnuo Wang
TL;DR
This work tackles the control gap in diffusion-based text-to-audio by introducing Audio Palette, a diffusion-transformer extension of Stable Audio Open that adds four time-varying acoustic controls (loudness, pitch, spectral centroid, and timbre) for Foley synthesis. It leverages LoRA-based parameter-efficient fine-tuning on a curated AudioSet Foley subset and introduces a three-scale classifier-free guidance mechanism to balance semantic fidelity, dynamics, and timbre. Empirical results show that Audio Palette achieves fine-grained, interpretable control with only modest degradation in audio quality and semantic alignment compared to a text-only baseline, while enabling more controllable and artist-friendly sound design. The paper also presents a scalable, modular pipeline emphasizing sequence conditioning and memory efficiency, fostering open-source development for controllable sound design in MIR and music/audio retrieval contexts.
Abstract
Recent advances in diffusion-based generative models have enabled high-quality text-to-audio synthesis, but fine-grained acoustic control remains a significant challenge in open-source research. We present Audio Palette, a diffusion transformer (DiT) based model that extends the Stable Audio Open architecture to address this "control gap" in controllable audio generation. Unlike prior approaches that rely solely on semantic conditioning, Audio Palette introduces four time-varying control signals, loudness, pitch, spectral centroid, and timbre, for precise and interpretable manipulation of acoustic features. The model is efficiently adapted for the nuanced domain of Foley synthesis using Low-Rank Adaptation (LoRA) on a curated subset of AudioSet, requiring only 0.85% of the original parameters to be trained. Experiments demonstrate that Audio Palette achieves fine-grained, interpretable control of sound attributes. Crucially, it accomplishes this novel controllability while maintaining high audio quality and strong semantic alignment to text prompts, with performance on standard metrics such as Frechet Audio Distance (FAD) and LAION-CLAP scores remaining comparable to the original baseline model. We provide a scalable, modular pipeline for audio research, emphasizing sequence-based conditioning, memory efficiency, and a three-scale classifier-free guidance mechanism for nuanced inference-time control. This work establishes a robust foundation for controllable sound design and performative audio synthesis in open-source settings, enabling a more artist-centric workflow in the broader context of music and sound information retrieval.