TADA! Tuning Audio Diffusion Models through Activation Steering
Audio diffusion models can synthesize high-fidelity music from text, yet their internal mechanisms for representing high-level concepts remain poorly understood. In this work, we use activation patching to demonstrate that distinct semantic musical concepts, such as the presence of specific instruments, vocals, or genre characteristics, are controlled by a small, shared subset of attention layers in state-of-the-art audio diffusion architectures. Next, we demonstrate that applying Contrastive Activation Addition and Sparse Autoencoders in these layers enables more precise control over the generated audio, indicating a direct benefit of the specialization phenomenon. By steering activations of the identified layers, we can alter specific musical elements with high precision, such as modulating tempo or changing a track's mood.
699281b350fb2c0be47838c8
Łukasz Staniszewski
claimed_verified
2026-02-17T15:53:25.346Z
63c7c19721bd95f80ed8ed80
/avatars/0b1c1ace991e0290118d4f99f619d809.svg
Lukasz Staniszewski
lukasz-staniszewski
user
lukasz-staniszewski
699281b350fb2c0be47838c9
Katarzyna Zaleska
699281b350fb2c0be47838ca
Mateusz Modrzejewski
699281b350fb2c0be47838cb
Kamil Deja
TADA! Tuning Audio Diffusion Models through Activation Steering
Audio diffusion models can synthesize high-fidelity music from text, yet their internal mechanisms for representing high-level concepts remain poorly understood. In this work, we use activation patching to demonstrate that distinct semantic musical concepts, such as the presence of specific instruments, vocals, or genre characteristics, are controlled by a small, shared subset of attention layers in state-of-the-art audio diffusion architectures. Next, we demonstrate that applying Contrastive Activation Addition and Sparse Autoencoders in these layers enables more precise control over the generated audio, indicating a direct benefit of the specialization phenomenon. By steering activations of the identified layers, we can alter specific musical elements with high precision, such as modulating tempo or changing a track's mood.
Research reveals that specific attention layers in audio diffusion models control distinct musical concepts, enabling precise manipulation of audio features through activation steering.
audio diffusion models attention layers activation patching Contrastive Activation Addition Sparse Autoencoders audio synthesis musical concepts semantic representation