MergeDNA: Context-aware Genome Modeling with Dynamic Tokenization through Token Merging
Modeling genomic sequences faces two unsolved challenges: the information density varies widely across different regions, while there is no clearly defined minimum vocabulary unit. Relying on either four primitive bases or independently designed DNA tokenizers, existing approaches with naive masked language modeling pre-training often fail to adapt to the varying complexities of genomic sequences. Leveraging Token Merging techniques, this paper introduces a hierarchical architecture that jointly optimizes a dynamic genomic tokenizer and latent Transformers with context-aware pre-training tasks. As for network structures, the tokenization module automatically chunks adjacent bases into words by stacking multiple layers of the differentiable token merging blocks with local-window constraints, then a Latent Encoder captures the global context of these merged words by full-attention blocks. Symmetrically employing a Latent Decoder and a Local Decoder, MergeDNA learns with two pre-training tasks: Merged Token Reconstruction simultaneously trains the dynamic tokenization module and adaptively filters important tokens, while Adaptive Masked Token Modeling learns to predict these filtered tokens to capture informative contents. Extensive experiments show that MergeDNA achieves superior performance on three popular DNA benchmarks and several multi-omics tasks with fine-tuning or zero-shot evaluation, outperforming typical tokenization methods and large-scale DNA foundation models.
6920670c8c38b39d6a482e4e
Siyuan Li
claimed_verified
2025-11-24T12:26:57.695Z
640f7083208821a59b74c757
https://cdn-avatars.huggingface.co/v1/production/uploads/1678735253848-640f7083208821a59b74c757.jpeg
Siyuan Li
Lupin1998
user
Lupin1998
6920670c8c38b39d6a482e4f
Kai Yu
6920670c8c38b39d6a482e50
Anna Wang
6920670c8c38b39d6a482e51
Zicheng Liu
6920670c8c38b39d6a482e52
Chang Yu
6920670c8c38b39d6a482e53
Jingbo Zhou
6920670c8c38b39d6a482e54
Qirong Yang
6920670c8c38b39d6a482e55
Yucheng Guo
6920670c8c38b39d6a482e56
Xiaoming Zhang
6920670c8c38b39d6a482e57
Stan Z. Li
MergeDNA: Context-aware Genome Modeling with Dynamic Tokenization through Token Merging
Modeling genomic sequences faces two unsolved challenges: the information density varies widely across different regions, while there is no clearly defined minimum vocabulary unit. Relying on either four primitive bases or independently designed DNA tokenizers, existing approaches with naive masked language modeling pre-training often fail to adapt to the varying complexities of genomic sequences. Leveraging Token Merging techniques, this paper introduces a hierarchical architecture that jointly optimizes a dynamic genomic tokenizer and latent Transformers with context-aware pre-training tasks. As for network structures, the tokenization module automatically chunks adjacent bases into words by stacking multiple layers of the differentiable token merging blocks with local-window constraints, then a Latent Encoder captures the global context of these merged words by full-attention blocks. Symmetrically employing a Latent Decoder and a Local Decoder, MergeDNA learns with two pre-training tasks: Merged Token Reconstruction simultaneously trains the dynamic tokenization module and adaptively filters important tokens, while Adaptive Masked Token Modeling learns to predict these filtered tokens to capture informative contents. Extensive experiments show that MergeDNA achieves superior performance on three popular DNA benchmarks and several multi-omics tasks with fine-tuning or zero-shot evaluation, outperforming typical tokenization methods and large-scale DNA foundation models.
MergeDNA uses a hierarchical architecture with Token Merging and latent Transformers to model genomic sequences, achieving superior performance on DNA benchmarks and multi-omics tasks.
Token Merging dynamic genomic tokenizer latent Transformers context-aware pre-training differentiable token merging blocks full-attention blocks Latent Encoder Latent Decoder Local Decoder Merged Token Reconstruction Adaptive Masked Token Modeling DNA benchmarks multi-omics tasks