Dynamic Large Concept Models: Latent Reasoning in an Adaptive Semantic Space
Large Language Models (LLMs) apply uniform computation to all tokens, despite language exhibiting highly non-uniform information density. This token-uniform regime wastes capacity on locally predictable spans while under-allocating computation to semantically critical transitions. We propose Dynamic Large Concept Models (DLCM), a hierarchical language modeling framework that learns semantic boundaries from latent representations and shifts computation from tokens to a compressed concept space where reasoning is more efficient. DLCM discovers variable-length concepts end-to-end without relying on predefined linguistic units. Hierarchical compression fundamentally changes scaling behavior. We introduce the first compression-aware scaling law, which disentangles token-level capacity, concept-level reasoning capacity, and compression ratio, enabling principled compute allocation under fixed FLOPs. To stably train this heterogeneous architecture, we further develop a decoupled μP parametrization that supports zero-shot hyperparameter transfer across widths and compression regimes. At a practical setting (R=4, corresponding to an average of four tokens per concept), DLCM reallocates roughly one-third of inference compute into a higher-capacity reasoning backbone, achieving a +2.69\% average improvement across 12 zero-shot benchmarks under matched inference FLOPs.
69573165832867f253525871
Xingwei Qu
claimed_verified
2026-01-02T15:38:07.662Z
646b4c9fdf2609a541c0866e
/avatars/7ec6c709017dcf32d4ac49d1e3820328.svg
Qu
ScottQu
user
ScottQu
69573165832867f253525872
Shaowen Wang
69573165832867f253525873
Zihao Huang
claimed_verified
2026-01-04T20:09:46.182Z
65a62085576772f531e13856
/avatars/72c67a60422e333ea4e323f7480ae0b7.svg
Huang Zihao
FetchFortune
user
FetchFortune
69573165832867f253525874
Kai Hua
claimed_verified
2026-01-02T15:37:53.467Z
64e851825ddcace745ba15bd
/avatars/7b6612c411222974d9ea181784eef915.svg
Kai Hua
kkish
user
kkish
69573165832867f253525875
Fan Yin
69573165832867f253525876
Rui-Jie Zhu
69573165832867f253525877
Jundong Zhou
69573165832867f253525878
Qiyang Min
69573165832867f253525879
Zihao Wang
69573165832867f25352587a
Yizhi Li
claimed_verified
2026-04-07T08:49:35.609Z
6382252f54421460665ec501
https://cdn-avatars.huggingface.co/v1/production/uploads/6382252f54421460665ec501/gW9fev3T5QPcNq4f9hqB1.jpeg
Yizhi Li
yizhilll
user
yizhilll
69573165832867f25352587b
Tianyu Zhang
69573165832867f25352587c
He Xing
69573165832867f25352587d
Zheng Zhang
69573165832867f25352587e
Yuxuan Song
69573165832867f25352587f
Tianyu Zheng
69573165832867f253525880
Zhiyuan Zeng
69573165832867f253525881
Chenghua Lin
69573165832867f253525882
Ge Zhang
claimed_verified
2026-01-12T10:36:46.402Z
638efcf4c67af472d316d424
/avatars/97a57859d7d87a3a8f1bb41d32a72bc2.svg
Ge Zhang
zhangysk
user
zhangysk
69573165832867f253525883
Wenhao Huang
Dynamic Large Concept Models: Latent Reasoning in an Adaptive Semantic Space
Large Language Models (LLMs) apply uniform computation to all tokens, despite language exhibiting highly non-uniform information density. This token-uniform regime wastes capacity on locally predictable spans while under-allocating computation to semantically critical transitions. We propose Dynamic Large Concept Models (DLCM), a hierarchical language modeling framework that learns semantic boundaries from latent representations and shifts computation from tokens to a compressed concept space where reasoning is more efficient. DLCM discovers variable-length concepts end-to-end without relying on predefined linguistic units. Hierarchical compression fundamentally changes scaling behavior. We introduce the first compression-aware scaling law, which disentangles token-level capacity, concept-level reasoning capacity, and compression ratio, enabling principled compute allocation under fixed FLOPs. To stably train this heterogeneous architecture, we further develop a decoupled μP parametrization that supports zero-shot hyperparameter transfer across widths and compression regimes. At a practical setting (R=4, corresponding to an average of four tokens per concept), DLCM reallocates roughly one-third of inference compute into a higher-capacity reasoning backbone, achieving a +2.69\% average improvement across 12 zero-shot benchmarks under matched inference FLOPs.