BOOM: Beyond Only One Modality KIT's Multimodal Multilingual Lecture Companion
The globalization of education and rapid growth of online learning have made localizing educational content a critical challenge. Lecture materials are inherently multimodal, combining spoken audio with visual slides, which requires systems capable of processing multiple input modalities. To provide an accessible and complete learning experience, translations must preserve all modalities: text for reading, slides for visual understanding, and speech for auditory learning. We present BOOM, a multimodal multilingual lecture companion that jointly translates lecture audio and slides to produce synchronized outputs across three modalities: translated text, localized slides with preserved visual elements, and synthesized speech. This end-to-end approach enables students to access lectures in their native language while aiming to preserve the original content in its entirety. Our experiments demonstrate that slide-aware transcripts also yield cascading benefits for downstream tasks such as summarization and question answering. We release our Slide Translation code at https://github.com/saikoneru/image-translator and integrate it in Lecture Translator at https://gitlab.kit.edu/kit/isl-ai4lt/lt-middleware/ltpipeline}\footnote{All released code and models are licensed under the MIT License.
692fedc026742347f61dadeb
Sai Koneru
claimed_verified
2025-12-04T08:49:01.935Z
63ea5f96b1d9c4ef71ed4da7
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/lZJbLglZgZEc9rPXgUn1Q.png
Sai Koneru
skoneru
user
skoneru
692fedc026742347f61dadec
Fabian Retkowski
692fedc026742347f61daded
Christian Huber
692fedc026742347f61dadee
Lukas Hilgert
692fedc026742347f61dadef
Seymanur Akti
692fedc026742347f61dadf0
Enes Yavuz Ugan
692fedc026742347f61dadf1
Alexander Waibel
692fedc026742347f61dadf2
Jan Niehues
BOOM: Beyond Only One Modality KIT's Multimodal Multilingual Lecture Companion
The globalization of education and rapid growth of online learning have made localizing educational content a critical challenge. Lecture materials are inherently multimodal, combining spoken audio with visual slides, which requires systems capable of processing multiple input modalities. To provide an accessible and complete learning experience, translations must preserve all modalities: text for reading, slides for visual understanding, and speech for auditory learning. We present BOOM, a multimodal multilingual lecture companion that jointly translates lecture audio and slides to produce synchronized outputs across three modalities: translated text, localized slides with preserved visual elements, and synthesized speech. This end-to-end approach enables students to access lectures in their native language while aiming to preserve the original content in its entirety. Our experiments demonstrate that slide-aware transcripts also yield cascading benefits for downstream tasks such as summarization and question answering. We release our Slide Translation code at https://github.com/saikoneru/image-translator and integrate it in Lecture Translator at https://gitlab.kit.edu/kit/isl-ai4lt/lt-middleware/ltpipeline}\footnote{All released code and models are licensed under the MIT License.
BOOM is a multimodal multilingual lecture companion that translates audio and slides, producing synchronized outputs across text, images, and speech, enhancing accessibility and preservation of educational content.
multimodal multilingual lecture companion slide-aware transcripts summarization question answering