Enhancing Linguistic Competence of Language Models through Pre-training with Language Learning Tasks
Language models (LMs) are pre-trained on raw text datasets to generate text sequences token-by-token. While this approach facilitates the learning of world knowledge and reasoning, it does not explicitly optimize for linguistic competence. To bridge this gap, we propose L2T, a pre-training framework integrating Language Learning Tasks alongside standard next-token prediction. Inspired by human language acquisition, L2T transforms raw text into structured input-output pairs to provide explicit linguistic stimulation. Pre-training LMs on a mixture of raw text and L2T data not only improves overall performance on linguistic competence benchmarks but accelerates its acquisition, while maintaining competitive performance on general reasoning tasks.
695f986a2450f142afb3c476
Atsuki Yamaguchi
claimed_verified
2026-01-09T08:36:12.874Z
63998cdd438fba22d95f0826
/avatars/20f04b6ad2d42361c2c7dfc540aa4694.svg
Atsuki Yamaguchi
atsuki-yamaguchi
user
atsuki-yamaguchi
695f986a2450f142afb3c477
Maggie Mi
695f986a2450f142afb3c478
Nikolaos Aletras
Enhancing Linguistic Competence of Language Models through Pre-training with Language Learning Tasks
Language models (LMs) are pre-trained on raw text datasets to generate text sequences token-by-token. While this approach facilitates the learning of world knowledge and reasoning, it does not explicitly optimize for linguistic competence. To bridge this gap, we propose L2T, a pre-training framework integrating Language Learning Tasks alongside standard next-token prediction. Inspired by human language acquisition, L2T transforms raw text into structured input-output pairs to provide explicit linguistic stimulation. Pre-training LMs on a mixture of raw text and L2T data not only improves overall performance on linguistic competence benchmarks but accelerates its acquisition, while maintaining competitive performance on general reasoning tasks.
Language models pre-trained with a framework combining standard next-token prediction and structured language learning tasks show enhanced linguistic competence without sacrificing general reasoning capabilities.
language models next-token prediction linguistic competence pre-training framework language learning tasks raw text structured input-output pairs linguistic stimulation