BidirLM: From Text to Omnimodal Bidirectional Encoders by Adapting and Composing Causal LLMs
Transforming causal generative language models into bidirectional encoders offers a powerful alternative to BERT-style architectures. However, current approaches remain limited: they lack consensus on optimal training objectives, suffer from catastrophic forgetting at scale, and fail to flexibly integrate the vast ecosystem of specialized generative models. In this work, through systematic ablations on the Gemma3 and Qwen3 families, we identify the key factors driving successful adaptation, highlighting the critical role of an often-omitted prior masking phase. To scale this process without original pre-training data, we introduce a dual strategy combining linear weight merging with a lightweight multi-domain data mixture that mitigates catastrophic forgetting. Finally, we augment our encoders by merging them with specialized causal models, seamlessly transferring modality- and domain-specific capabilities. This open-source recipe, designed for any causal decoder LLM, yields BidirLM, a family of five encoders that outperform alternatives on text, vision, and audio representation benchmarks.
69d16d472c523d36431faa48
Nicolas Boizard
claimed_verified
2026-04-07T14:02:17.017Z
62be186a5f59ff2320e6e32b
https://cdn-avatars.huggingface.co/v1/production/uploads/62be186a5f59ff2320e6e32b/W_emoC2uItM-MJZyCfIKI.png
Nicolas-BZRD
Nicolas-BZRD
user
Nicolas-BZRD
69d16d472c523d36431faa49
Théo Deschamps-Berger
claimed_verified
2026-04-08T08:06:31.664Z
64b8e1d256f81f15e2a6c625
https://cdn-avatars.huggingface.co/v1/production/uploads/64b8e1d256f81f15e2a6c625/N-0Ia96QpJBzgs8JytGjH.jpeg
Théo Deschamps-Berger
TheoDB
user
TheoDB
69d16d472c523d36431faa4a
Hippolyte Gisserot-Boukhlef
69d16d472c523d36431faa4b
Céline Hudelot
69d16d472c523d36431faa4c
Pierre Colombo
BidirLM: From Text to Omnimodal Bidirectional Encoders by Adapting and Composing Causal LLMs
Transforming causal generative language models into bidirectional encoders offers a powerful alternative to BERT-style architectures. However, current approaches remain limited: they lack consensus on optimal training objectives, suffer from catastrophic forgetting at scale, and fail to flexibly integrate the vast ecosystem of specialized generative models. In this work, through systematic ablations on the Gemma3 and Qwen3 families, we identify the key factors driving successful adaptation, highlighting the critical role of an often-omitted prior masking phase. To scale this process without original pre-training data, we introduce a dual strategy combining linear weight merging with a lightweight multi-domain data mixture that mitigates catastrophic forgetting. Finally, we augment our encoders by merging them with specialized causal models, seamlessly transferring modality- and domain-specific capabilities. This open-source recipe, designed for any causal decoder LLM, yields BidirLM, a family of five encoders that outperform alternatives on text, vision, and audio representation benchmarks.
Adapting causal generative language models into bidirectional encoders through systematic ablation and novel merging strategies achieves superior performance across multiple modalities.
causal generative language models bidirectional encoders BERT-style architectures catastrophic forgetting prior masking phase linear weight merging multi-domain data mixture specialized generative models causal decoder LLM BidirLM