Mecellem Models: Turkish Models Trained from Scratch and Continually Pre-trained for the Legal Domain
This paper presents Mecellem models, a framework for developing specialized language models for the Turkish legal domain through domain adaptation strategies. We make two contributions: (1)Encoder Model Pre-trained from Scratch: ModernBERT-based bidirectional encoders pre-trained on a Turkish-dominant corpus of 112.7 billion tokens. We implement a checkpoint selection strategy that evaluates downstream retrieval performance throughout training, revealing that optimal checkpoints achieve best retrieval scores before pre-training loss reaches its minimum. Our encoder models achieve top-3 rankings on the Turkish retrieval leaderboard, with smaller models (155M parameters) achieving comparable performance to larger reference models (307M-567M parameters). Our approach achieves 92.36% production efficiency compared to state-of-the-art models (embeddinggemma-300m: 100.00%, BAAI/bge-m3: 99.54%, newmindai/bge-m3-stsb: 94.38%), ranking fourth overall despite requiring less computational resources. SOTA models rely on multi-stage, computationally intensive training pipelines, making our single-stage pre-training followed by efficient post-training approach a cost-effective alternative; (2)Decoder Model with Continual Pre-training (CPT): Qwen3-1.7B and Qwen3-4B models adapted to Turkish legal domain through controlled curriculum learning. Four-phase CPT with optimal sample ratios enables gradual transition from general language knowledge to specialized legal terminology and long-context reasoning. This approach achieves 36.2% perplexity reduction on Turkish legal text, demonstrating domain adaptation gains.
6973628a5f9d15c14e26fd23
Özgür Uğur
claimed_verified
2026-01-26T08:32:42.505Z
66504e2b54eda33d54155874
https://cdn-avatars.huggingface.co/v1/production/uploads/66504e2b54eda33d54155874/5FypaPjRdZPCABqWbZV4k.jpeg
Özgür UĞUR
zgrgr
user
zgrgr
6973628a5f9d15c14e26fd24
Mahmut Göksu
6973628a5f9d15c14e26fd25
Mahmut Çimen
6973628a5f9d15c14e26fd26
Musa Yılmaz
claimed_verified
2026-01-27T09:44:02.593Z
653b9d1f01c329e22bfd3038
/avatars/8468327bfdacd259bf684a0d9f2b0816.svg
Musa YILMAZ
yilmazmusa-ml
user
yilmazmusa-ml
6973628a5f9d15c14e26fd27
Esra Şavirdi
6973628a5f9d15c14e26fd28
Alp Talha Demir
6973628a5f9d15c14e26fd29
Rumeysa Güllüce
6973628a5f9d15c14e26fd2a
İclal Çetin
6973628a5f9d15c14e26fd2b
Ömer Can Sağbaş
Mecellem Models: Turkish Models Trained from Scratch and Continually Pre-trained for the Legal Domain
This paper presents Mecellem models, a framework for developing specialized language models for the Turkish legal domain through domain adaptation strategies. We make two contributions: (1)Encoder Model Pre-trained from Scratch: ModernBERT-based bidirectional encoders pre-trained on a Turkish-dominant corpus of 112.7 billion tokens. We implement a checkpoint selection strategy that evaluates downstream retrieval performance throughout training, revealing that optimal checkpoints achieve best retrieval scores before pre-training loss reaches its minimum. Our encoder models achieve top-3 rankings on the Turkish retrieval leaderboard, with smaller models (155M parameters) achieving comparable performance to larger reference models (307M-567M parameters). Our approach achieves 92.36% production efficiency compared to state-of-the-art models (embeddinggemma-300m: 100.00%, BAAI/bge-m3: 99.54%, newmindai/bge-m3-stsb: 94.38%), ranking fourth overall despite requiring less computational resources. SOTA models rely on multi-stage, computationally intensive training pipelines, making our single-stage pre-training followed by efficient post-training approach a cost-effective alternative; (2)Decoder Model with Continual Pre-training (CPT): Qwen3-1.7B and Qwen3-4B models adapted to Turkish legal domain through controlled curriculum learning. Four-phase CPT with optimal sample ratios enables gradual transition from general language knowledge to specialized legal terminology and long-context reasoning. This approach achieves 36.2% perplexity reduction on Turkish legal text, demonstrating domain adaptation gains.
A framework for developing specialized Turkish legal language models through domain adaptation, featuring a pre-trained encoder model and decoder models with continual pre-training for enhanced legal text processing.
domain adaptation pre-trained models encoder models decoder models continual pre-training curriculum learning perplexity tokenization parameter-efficient fine-tuning retrieval performance downstream tasks multi-stage training single-stage pre-training post-training embedding models legal domain Turkish language BERT-based encoders Qwen3 models controlled curriculum learning sample ratios long-context reasoning