SAM Audio: Segment Anything in Audio
General audio source separation is a key capability for multimodal AI systems that can perceive and reason about sound. Despite substantial progress in recent years, existing separation models are either domain-specific, designed for fixed categories such as speech or music, or limited in controllability, supporting only a single prompting modality such as text. In this work, we present SAM Audio, a foundation model for general audio separation that unifies text, visual, and temporal span prompting within a single framework. Built on a diffusion transformer architecture, SAM Audio is trained with flow matching on large-scale audio data spanning speech, music, and general sounds, and can flexibly separate target sources described by language, visual masks, or temporal spans. The model achieves state-of-the-art performance across a diverse suite of benchmarks, including general sound, speech, music, and musical instrument separation in both in-the-wild and professionally produced audios, substantially outperforming prior general-purpose and specialized systems. Furthermore, we introduce a new real-world separation benchmark with human-labeled multimodal prompts and a reference-free evaluation model that correlates strongly with human judgment.
694aa9bcd8d7445f420c171d
Bowen Shi
694aa9bcd8d7445f420c171e
Andros Tjandra
694aa9bcd8d7445f420c171f
John Hoffman
694aa9bcd8d7445f420c1720
Helin Wang
694aa9bcd8d7445f420c1721
Yi-Chiao Wu
694aa9bcd8d7445f420c1722
Luya Gao
694aa9bcd8d7445f420c1723
Julius Richter
694aa9bcd8d7445f420c1724
Matt Le
694aa9bcd8d7445f420c1725
Apoorv Vyas
claimed_verified
2025-12-31T20:57:54.500Z
638697308b5acae8d251cdec
/avatars/dcfc47ee012439cc5ecc28e48ff26983.svg
Apoorv Vyas
apoorv2904
user
apoorv2904
694aa9bcd8d7445f420c1726
Sanyuan Chen
694aa9bcd8d7445f420c1727
Christoph Feichtenhofer
694aa9bcd8d7445f420c1728
Piotr Dollár
694aa9bcd8d7445f420c1729
Wei-Ning Hsu
694aa9bcd8d7445f420c172a
Ann Lee
SAM Audio: Segment Anything in Audio
General audio source separation is a key capability for multimodal AI systems that can perceive and reason about sound. Despite substantial progress in recent years, existing separation models are either domain-specific, designed for fixed categories such as speech or music, or limited in controllability, supporting only a single prompting modality such as text. In this work, we present SAM Audio, a foundation model for general audio separation that unifies text, visual, and temporal span prompting within a single framework. Built on a diffusion transformer architecture, SAM Audio is trained with flow matching on large-scale audio data spanning speech, music, and general sounds, and can flexibly separate target sources described by language, visual masks, or temporal spans. The model achieves state-of-the-art performance across a diverse suite of benchmarks, including general sound, speech, music, and musical instrument separation in both in-the-wild and professionally produced audios, substantially outperforming prior general-purpose and specialized systems. Furthermore, we introduce a new real-world separation benchmark with human-labeled multimodal prompts and a reference-free evaluation model that correlates strongly with human judgment.
SAM Audio, a diffusion transformer-based foundation model, achieves superior performance in general audio separation using unified text, visual, and temporal span prompts across various audio types.
diffusion transformer architecture flow matching audio separation general sound speech music musical instrument separation in-the-wild audio professionally produced audio real-world separation benchmark human-labeled multimodal prompts reference-free evaluation model