Advancing Polish Language Modeling through Tokenizer Optimization in the Bielik v3 7B and 11B Series
The development of the Bielik v3 PL series, encompassing both the 7B and 11B parameter variants, represents a significant milestone in the field of language-specific large language model (LLM) optimization. While general-purpose models often demonstrate impressive multilingual capabilities, they frequently suffer from a fundamental architectural inefficiency: the use of universal tokenizers. These tokenizers, typically designed to cover a broad spectrum of languages, often fail to capture the morphological nuances of specific languages like Polish, leading to higher fertility ratios, increased inference costs, and restricted effective context windows. This report details the transition from the universal Mistral-based tokenization to a dedicated Polish-optimized vocabulary for the Bielik v3 models, exploring the FOCUS-based embedding initialization, the multi-stage pretraining curriculum, and the subsequent post-training alignment involving Supervised Fine-Tuning, Direct Preference Optimization, and Reinforcement Learning through Group Relative Policy Optimization with verifiable rewards.
69ddc8c3aea31e227c22c5e9
Krzysztof Ociepa
69ddc8c3aea31e227c22c5ea
Łukasz Flis
69ddc8c3aea31e227c22c5eb
Remigiusz Kinas
69ddc8c3aea31e227c22c5ec
Krzysztof Wróbel
69ddc8c3aea31e227c22c5ed
Adrian Gwoździej
Advancing Polish Language Modeling through Tokenizer Optimization in the Bielik v3 7B and 11B Series
The development of the Bielik v3 PL series, encompassing both the 7B and 11B parameter variants, represents a significant milestone in the field of language-specific large language model (LLM) optimization. While general-purpose models often demonstrate impressive multilingual capabilities, they frequently suffer from a fundamental architectural inefficiency: the use of universal tokenizers. These tokenizers, typically designed to cover a broad spectrum of languages, often fail to capture the morphological nuances of specific languages like Polish, leading to higher fertility ratios, increased inference costs, and restricted effective context windows. This report details the transition from the universal Mistral-based tokenization to a dedicated Polish-optimized vocabulary for the Bielik v3 models, exploring the FOCUS-based embedding initialization, the multi-stage pretraining curriculum, and the subsequent post-training alignment involving Supervised Fine-Tuning, Direct Preference Optimization, and Reinforcement Learning through Group Relative Policy Optimization with verifiable rewards.
The Bielik v3 PL series achieves improved language-specific performance through specialized Polish tokenization, FOCUS-based embeddings, and multi-stage training with supervised fine-tuning, direct preference optimization, and reinforcement learning.
tokenization FOCUS-based embedding initialization multi-stage pretraining curriculum post-training alignment Supervised Fine-Tuning Direct Preference Optimization Reinforcement Learning Group Relative Policy Optimization verifiable rewards