HI-TransPA: Hearing Impairments Translation Personal Assistant
To provide a unified and flexible solution for daily communication among hearing-impaired individuals, we introduce the Omni-Model paradigm into assistive technology and present HI-TransPA, an instruction-driven audio-visual personal assistant. The model fuses indistinct speech with high-frame-rate lip dynamics, enabling both translation and dialogue within a single multimodal framework. To tackle the challenges of noisy and heterogeneous raw data and the limited adaptability of existing Omni-Models to hearing-impaired speech, we construct a comprehensive preprocessing and curation pipeline that detects facial landmarks, isolates and stabilizes the lip region, and quantitatively assesses multimodal sample quality. These quality scores guide a curriculum learning strategy that first trains on clean, high-confidence samples and progressively incorporates harder cases to strengthen model robustness. We further adopt a SigLIP encoder combined with a Unified 3D-Resampler to efficiently encode high-frame-rate lip motion. Experiments on our purpose-built HI-Dialogue dataset show that HI-TransPA achieves state-of-the-art performance in both literal accuracy and semantic fidelity. This work establishes a foundation for applying Omni-Models to assistive communication technology, providing an end-to-end modeling framework and essential processing tools for future research.
691700dab15c6ccd9319525e
Zhiming Ma
claimed_verified
2025-11-17T10:31:37.452Z
64a0ed5ed5374ca472cfb0ac
https://cdn-avatars.huggingface.co/v1/production/uploads/64a0ed5ed5374ca472cfb0ac/n_wXamXfR_PPn0hRbnR1X.jpeg
ZhimingMa
JimmyMa99
user
JimmyMa99
691700dab15c6ccd9319525f
Shiyu Gan
691700dab15c6ccd93195260
Junhao Zhao
691700dab15c6ccd93195261
Xianming Li
691700dab15c6ccd93195262
Qingyun Pan
691700dab15c6ccd93195263
Peidong Wang
691700dab15c6ccd93195264
Mingjun Pan
691700dab15c6ccd93195265
Yuhao Mo
691700dab15c6ccd93195266
Jiajie Cheng
691700dab15c6ccd93195267
Chengxin Chen
691700dab15c6ccd93195268
Zhonglun Cao
691700dab15c6ccd93195269
Chonghan Liu
691700dab15c6ccd9319526a
Shi Cheng
HI-TransPA: Hearing Impairments Translation Personal Assistant
To provide a unified and flexible solution for daily communication among hearing-impaired individuals, we introduce the Omni-Model paradigm into assistive technology and present HI-TransPA, an instruction-driven audio-visual personal assistant. The model fuses indistinct speech with high-frame-rate lip dynamics, enabling both translation and dialogue within a single multimodal framework. To tackle the challenges of noisy and heterogeneous raw data and the limited adaptability of existing Omni-Models to hearing-impaired speech, we construct a comprehensive preprocessing and curation pipeline that detects facial landmarks, isolates and stabilizes the lip region, and quantitatively assesses multimodal sample quality. These quality scores guide a curriculum learning strategy that first trains on clean, high-confidence samples and progressively incorporates harder cases to strengthen model robustness. We further adopt a SigLIP encoder combined with a Unified 3D-Resampler to efficiently encode high-frame-rate lip motion. Experiments on our purpose-built HI-Dialogue dataset show that HI-TransPA achieves state-of-the-art performance in both literal accuracy and semantic fidelity. This work establishes a foundation for applying Omni-Models to assistive communication technology, providing an end-to-end modeling framework and essential processing tools for future research.
HI-TransPA, an instruction-driven audio-visual personal assistant, uses Omni-Model paradigm to translate and dialogue by fusing speech with lip dynamics, achieving state-of-the-art performance in assistive communication for hearing-impaired individuals.
Omni-Model HI-TransPA instruction-driven audio-visual personal assistant indistinct speech high-frame-rate lip dynamics preprocessing curation pipeline facial landmarks lip region stabilization multimodal sample quality curriculum learning SigLIP encoder Unified 3D-Resampler HI-Dialogue dataset literal accuracy semantic fidelity