Fair splits flip the leaderboard: CHANRG reveals limited generalization in RNA secondary-structure prediction
Accurate prediction of RNA secondary structure underpins transcriptome annotation, mechanistic analysis of non-coding RNAs, and RNA therapeutic design. Recent gains from deep learning and RNA foundation models are difficult to interpret because current benchmarks may overestimate generalization across RNA families. We present the Comprehensive Hierarchical Annotation of Non-coding RNA Groups (CHANRG), a benchmark of 170{,}083 structurally non-redundant RNAs curated from more than 10 million sequences in Rfam~15.0 using structure-aware deduplication, genome-aware split design and multiscale structural evaluation. Across 29 predictors, foundation-model methods achieved the highest held-out accuracy but lost most of that advantage out of distribution, whereas structured decoders and direct neural predictors remained markedly more robust. This gap persisted after controlling for sequence length and reflected both loss of structural coverage and incorrect higher-order wiring. Together, CHANRG and a padding-free, symmetry-aware evaluation stack provide a stricter and batch-invariant framework for developing RNA structure predictors with demonstrable out-of-distribution robustness.
69c358a295b48f16a300781b
Zhiyuan Chen
claimed_verified
2026-03-25T10:23:21.932Z
642687f7057dd1732ca4c55b
https://cdn-avatars.huggingface.co/v1/production/uploads/642687f7057dd1732ca4c55b/Ahz8yDb4vYU3P0XDM0cOB.png
Zhiyuan Chen
ZhiyuanChen
user
ZhiyuanChen
69c358a295b48f16a300781c
Zhenfeng Deng
69c358a295b48f16a300781d
Pan Deng
69c358a295b48f16a300781e
Yue Liao
69c358a295b48f16a300781f
Xiu Su
69c358a295b48f16a3007820
Peng Ye
69c358a295b48f16a3007821
Xihui Liu
Fair splits flip the leaderboard: CHANRG reveals limited generalization in RNA secondary-structure prediction
Accurate prediction of RNA secondary structure underpins transcriptome annotation, mechanistic analysis of non-coding RNAs, and RNA therapeutic design. Recent gains from deep learning and RNA foundation models are difficult to interpret because current benchmarks may overestimate generalization across RNA families. We present the Comprehensive Hierarchical Annotation of Non-coding RNA Groups (CHANRG), a benchmark of 170{,}083 structurally non-redundant RNAs curated from more than 10 million sequences in Rfam~15.0 using structure-aware deduplication, genome-aware split design and multiscale structural evaluation. Across 29 predictors, foundation-model methods achieved the highest held-out accuracy but lost most of that advantage out of distribution, whereas structured decoders and direct neural predictors remained markedly more robust. This gap persisted after controlling for sequence length and reflected both loss of structural coverage and incorrect higher-order wiring. Together, CHANRG and a padding-free, symmetry-aware evaluation stack provide a stricter and batch-invariant framework for developing RNA structure predictors with demonstrable out-of-distribution robustness.
CHANRG benchmark evaluates RNA structure prediction models with improved rigor, revealing that foundation models lose robustness outside distribution despite higher held-out accuracy.
RNA secondary structure deep learning RNA foundation models benchmark structure-aware deduplication genome-aware split design multiscale structural evaluation out-of-distribution robustness structured decoders direct neural predictors padding-free evaluation symmetry-aware evaluation