LM-Lexicon: Improving Definition Modeling via Harmonizing Semantic Experts
We introduce LM-Lexicon, an innovative definition modeling approach that incorporates data clustering, semantic expert learning, and model merging using a sparse mixture-of-experts architecture. By decomposing the definition modeling task into specialized semantic domains, where small language models are trained as domain experts, LM-Lexicon achieves substantial improvements (+7% BLEU score compared with the prior state-of-the-art model) over existing methods on five widely used benchmarks. Empirically, we demonstrate that 1) the clustering strategy enables fine-grained expert specialization with nearly 10% improvement in definition quality; 2) the semantic-aware domain-level routing mechanism achieves higher expert efficacy (+1%) than conventional token-level routing; and 3) further performance gains can be obtained through test-time compute and semantic expert scaling. Our work advances definition modeling while providing insights into the development of efficient language models for semantic-intensive applications.
6994768e046aed98c9b8fafb
Yang Liu
claimed_verified
2026-02-17T15:08:23.567Z
6191cc9e6d34e827404cebab
https://cdn-avatars.huggingface.co/v1/production/uploads/1674119843175-6191cc9e6d34e827404cebab.jpeg
Yang
jacklanda
user
jacklanda
6994768e046aed98c9b8fafc
Jiaye Yang
6994768e046aed98c9b8fafd
Weikang Li
6994768e046aed98c9b8fafe
Jiahui Liang
6994768e046aed98c9b8faff
Yang Li
6994768e046aed98c9b8fb00
Lingyong Yan
LM-Lexicon: Improving Definition Modeling via Harmonizing Semantic Experts
We introduce LM-Lexicon, an innovative definition modeling approach that incorporates data clustering, semantic expert learning, and model merging using a sparse mixture-of-experts architecture. By decomposing the definition modeling task into specialized semantic domains, where small language models are trained as domain experts, LM-Lexicon achieves substantial improvements (+7% BLEU score compared with the prior state-of-the-art model) over existing methods on five widely used benchmarks. Empirically, we demonstrate that 1) the clustering strategy enables fine-grained expert specialization with nearly 10% improvement in definition quality; 2) the semantic-aware domain-level routing mechanism achieves higher expert efficacy (+1%) than conventional token-level routing; and 3) further performance gains can be obtained through test-time compute and semantic expert scaling. Our work advances definition modeling while providing insights into the development of efficient language models for semantic-intensive applications.
LM-Lexicon improves definition modeling through data clustering, semantic expert learning, and sparse mixture-of-experts architecture, achieving higher BLEU scores and better expert specialization.
definition modeling data clustering semantic expert learning model merging sparse mixture-of-experts architecture domain experts semantic domains language models expert specialization semantic-aware domain-level routing