HeartMuLa: A Family of Open Sourced Music Foundation Models
We present a family of open-source Music Foundation Models designed to advance large-scale music understanding and generation across diverse tasks and modalities. Our framework consists of four major components: (1) HeartCLAP, an audio-text alignment model; (2) HeartTranscriptor, a robust lyric recognition model optimized for real-world music scenarios; and (3) HeartCodec, a low-frame-rate (12.5 Hz) yet high-fidelity music codec tokenizer that captures long-range musical structure while preserving fine-grained acoustic details and enabling efficient autoregressive modeling; (4) HeartMuLa, an LLM-based song generation model capable of synthesizing high-fidelity music under rich, user-controllable conditions (e.g., textual style descriptions, lyrics, and reference audio). In addition, it provides two specialized modes: (i) fine-grained musical attribute control, which allows users to specify the style of different song sections (e.g., intro, verse, chorus) using natural language prompts; and (ii) short, engaging music generation, which is suitable as background music for short videos. Lastly, HeartMuLa improves significantly when scaled to 7B parameters. For the first time, we show that a Suno-level, commercial-grade system can be reproduced using academic-scale data and GPU resources. We expect these foundation models to serve as strong baselines for future research and to facilitate practical applications in multimodal content production.
6969e2dc32f0333869ff952e
Dongchao Yang
admin_assigned
2026-01-16T15:43:00.669Z
63c7636b656e7822e23e6f6b
/avatars/41bfd5e1ce6daab6058eacfd33c7a268.svg
Dongchao Yang
Dongchao
user
Dongchao
6969e2dc32f0333869ff952f
Yuxin Xie
admin_assigned
2026-01-16T15:43:07.620Z
66a1983418ce1e6e432fc0d3
/avatars/5f2fd2427057eaf1de5c3c2ce6d19c22.svg
Yuxin XIE
bverxie
user
bverxie
6969e2dc32f0333869ff9530
Yuguo Yin
6969e2dc32f0333869ff9531
Zheyu Wang
6969e2dc32f0333869ff9532
Xiaoyu Yi
claimed_verified
2026-01-19T09:23:44.783Z
6835b5e573013ebe62da0179
/avatars/baa28d0f7d76d346e6204e8407ccab42.svg
fish yi
f1shy1
user
f1shy1
6969e2dc32f0333869ff9533
Gongxi Zhu
admin_assigned
2026-01-16T15:43:33.999Z
643de56c0efe9e9de09a720e
/avatars/1911ee9f28a328f169b7fcb380890739.svg
Gongxi Zhu
Gongxizhu
user
Gongxizhu
6969e2dc32f0333869ff9534
Xiaolong Weng
6969e2dc32f0333869ff9535
Zihan Xiong
6969e2dc32f0333869ff9536
Yingzhe Ma
6969e2dc32f0333869ff9537
Dading Cong
6969e2dc32f0333869ff9538
Jingliang Liu
6969e2dc32f0333869ff9539
Zihang Huang
6969e2dc32f0333869ff953a
Jinghan Ru
6969e2dc32f0333869ff953b
Rongjie Huang
6969e2dc32f0333869ff953c
Haoran Wan
6969e2dc32f0333869ff953d
Peixu Wang
6969e2dc32f0333869ff953e
Kuoxi Yu
6969e2dc32f0333869ff953f
Helin Wang
6969e2dc32f0333869ff9540
Liming Liang
6969e2dc32f0333869ff9541
Xianwei Zhuang
6969e2dc32f0333869ff9542
Yuanyuan Wang
admin_assigned
2026-01-16T15:44:41.696Z
64e85f3572c2b9493fc09fcb
/avatars/673626e7e0c7fdf2e5e6e98422cdad3d.svg
Wang
YuanyuanWang
user
YuanyuanWang
6969e2dc32f0333869ff9543
Haohan Guo
admin_assigned
2026-01-16T15:44:26.552Z
63783d9d84318944acd305c4
/avatars/cafd4fd0287e77cc6ebf37c8c8509174.svg
Haohan Guo
hhguo
user
hhguo
6969e2dc32f0333869ff9544
Junjie Cao
6969e2dc32f0333869ff9545
Zeqian Ju
6969e2dc32f0333869ff9546
Songxiang Liu
6969e2dc32f0333869ff9547
Yuewen Cao
6969e2dc32f0333869ff9548
Heming Weng
6969e2dc32f0333869ff9549
Yuexian Zou
HeartMuLa: A Family of Open Sourced Music Foundation Models
We present a family of open-source Music Foundation Models designed to advance large-scale music understanding and generation across diverse tasks and modalities. Our framework consists of four major components: (1) HeartCLAP, an audio-text alignment model; (2) HeartTranscriptor, a robust lyric recognition model optimized for real-world music scenarios; and (3) HeartCodec, a low-frame-rate (12.5 Hz) yet high-fidelity music codec tokenizer that captures long-range musical structure while preserving fine-grained acoustic details and enabling efficient autoregressive modeling; (4) HeartMuLa, an LLM-based song generation model capable of synthesizing high-fidelity music under rich, user-controllable conditions (e.g., textual style descriptions, lyrics, and reference audio). In addition, it provides two specialized modes: (i) fine-grained musical attribute control, which allows users to specify the style of different song sections (e.g., intro, verse, chorus) using natural language prompts; and (ii) short, engaging music generation, which is suitable as background music for short videos. Lastly, HeartMuLa improves significantly when scaled to 7B parameters. For the first time, we show that a Suno-level, commercial-grade system can be reproduced using academic-scale data and GPU resources. We expect these foundation models to serve as strong baselines for future research and to facilitate practical applications in multimodal content production.
A suite of open-source music foundation models is introduced, featuring components for audio-text alignment, lyric recognition, music coding, and large language model-based song generation with controllable attributes and scalable parameterization.
Music Foundation Models audio-text alignment model lyric recognition model music codec tokenizer LLM-based song generation model autoregressive modeling musical attribute control short video background music generation parameter scaling