Proactive Hearing Assistants that Isolate Egocentric Conversations
We introduce proactive hearing assistants that automatically identify and separate the wearer's conversation partners, without requiring explicit prompts. Our system operates on egocentric binaural audio and uses the wearer's self-speech as an anchor, leveraging turn-taking behavior and dialogue dynamics to infer conversational partners and suppress others. To enable real-time, on-device operation, we propose a dual-model architecture: a lightweight streaming model runs every 12.5 ms for low-latency extraction of the conversation partners, while a slower model runs less frequently to capture longer-range conversational dynamics. Results on real-world 2- and 3-speaker conversation test sets, collected with binaural egocentric hardware from 11 participants totaling 6.8 hours, show generalization in identifying and isolating conversational partners in multi-conversation settings. Our work marks a step toward hearing assistants that adapt proactively to conversational dynamics and engagement. More information can be found on our website: https://proactivehearing.cs.washington.edu/
691d88c30247f3d258ef333a
Guilin Hu
claimed_verified
2025-11-19T09:12:34.228Z
68b8e1f1103c1ffe110dc9b6
/avatars/d3d27e24a90af3814317757f4dfb6194.svg
Guilin Hu
guilinhu
user
guilinhu
691d88c30247f3d258ef333b
Malek Itani
691d88c30247f3d258ef333c
Tuochao Chen
691d88c30247f3d258ef333d
Shyamnath Gollakota
Proactive Hearing Assistants that Isolate Egocentric Conversations
We introduce proactive hearing assistants that automatically identify and separate the wearer's conversation partners, without requiring explicit prompts. Our system operates on egocentric binaural audio and uses the wearer's self-speech as an anchor, leveraging turn-taking behavior and dialogue dynamics to infer conversational partners and suppress others. To enable real-time, on-device operation, we propose a dual-model architecture: a lightweight streaming model runs every 12.5 ms for low-latency extraction of the conversation partners, while a slower model runs less frequently to capture longer-range conversational dynamics. Results on real-world 2- and 3-speaker conversation test sets, collected with binaural egocentric hardware from 11 participants totaling 6.8 hours, show generalization in identifying and isolating conversational partners in multi-conversation settings. Our work marks a step toward hearing assistants that adapt proactively to conversational dynamics and engagement. More information can be found on our website: https://proactivehearing.cs.washington.edu/
A proactive hearing assistant system identifies and separates conversation partners in real-time using a dual-model architecture on binaural audio, adapting to conversational dynamics without explicit prompts.
egocentric binaural audio self-speech turn-taking behavior dialogue dynamics dual-model architecture lightweight streaming model real-time on-device operation conversational partners suppression generalization multi-conversation settings