AudioSAE: Towards Understanding of Audio-Processing Models with Sparse AutoEncoders
Sparse Autoencoders (SAEs) are powerful tools for interpreting neural representations, yet their use in audio remains underexplored. We train SAEs across all encoder layers of Whisper and HuBERT, provide an extensive evaluation of their stability, interpretability, and show their practical utility. Over 50% of the features remain consistent across random seeds, and reconstruction quality is preserved. SAE features capture general acoustic and semantic information as well as specific events, including environmental noises and paralinguistic sounds (e.g. laughter, whispering) and disentangle them effectively, requiring removal of only 19-27% of features to erase a concept. Feature steering reduces Whisper's false speech detections by 70% with negligible WER increase, demonstrating real-world applicability. Finally, we find SAE features correlated with human EEG activity during speech perception, indicating alignment with human neural processing. The code and checkpoints are available at https://github.com/audiosae/audiosae_demo.
69871a872d626112378ad69f
Georgii Aparin
claimed_verified
2026-02-09T08:34:22.635Z
660fd34df03515e4ff3f2b64
/avatars/0c2a29b1081ece881234acdd8ef9371a.svg
Georgii Aparin
Egorgij21
user
Egorgij21
69871a872d626112378ad6a0
Tasnima Sadekova
69871a872d626112378ad6a1
Alexey Rukhovich
69871a872d626112378ad6a2
Assel Yermekova
69871a872d626112378ad6a3
Laida Kushnareva
69871a872d626112378ad6a4
Vadim Popov
69871a872d626112378ad6a5
Kristian Kuznetsov
69871a872d626112378ad6a6
Irina Piontkovskaya
AudioSAE: Towards Understanding of Audio-Processing Models with Sparse AutoEncoders
Sparse Autoencoders (SAEs) are powerful tools for interpreting neural representations, yet their use in audio remains underexplored. We train SAEs across all encoder layers of Whisper and HuBERT, provide an extensive evaluation of their stability, interpretability, and show their practical utility. Over 50% of the features remain consistent across random seeds, and reconstruction quality is preserved. SAE features capture general acoustic and semantic information as well as specific events, including environmental noises and paralinguistic sounds (e.g. laughter, whispering) and disentangle them effectively, requiring removal of only 19-27% of features to erase a concept. Feature steering reduces Whisper's false speech detections by 70% with negligible WER increase, demonstrating real-world applicability. Finally, we find SAE features correlated with human EEG activity during speech perception, indicating alignment with human neural processing. The code and checkpoints are available at https://github.com/audiosae/audiosae_demo.
Sparse Autoencoders trained on Whisper and HuBERT models demonstrate stable feature extraction and effective disentanglement of acoustic and semantic information, showing practical applications in audio processing and correlation with human neural activity.
Sparse Autoencoders encoder layers Whisper HuBERT feature steering false speech detections WER EEG activity speech perception