Avey-B
Compact pretrained bidirectional encoders remain the backbone of industrial NLP under tight compute and memory budgets. Their effectiveness stems from self-attention's ability to deliver high-quality bidirectional contextualization with sequence-level parallelism, as popularized by BERT-style architectures. Recently, Avey was introduced as an autoregressive, attention-free alternative that naturally admits an encoder-only adaptation. In this paper, we reformulate Avey for the encoder-only paradigm and propose several innovations to its architecture, including decoupled static and dynamic parameterizations, stability-oriented normalization, and neural compression. Results show that this reformulated architecture compares favorably to four widely used Transformer-based encoders, consistently outperforming them on standard token-classification and information-retrieval benchmarks while scaling more efficiently to long contexts.
69956b858d17d1ee8c10ed21
Devang Acharya
claimed_verified
2026-02-19T09:53:09.096Z
683c849152d744770fe82ade
https://cdn-avatars.huggingface.co/v1/production/uploads/683c849152d744770fe82ade/C_TMUIF5PJ1xkxwKxpBFl.jpeg
Rain
rainnekoneko
user
rainnekoneko
69956b858d17d1ee8c10ed22
Mohammad Hammoud
Avey-B
Compact pretrained bidirectional encoders remain the backbone of industrial NLP under tight compute and memory budgets. Their effectiveness stems from self-attention's ability to deliver high-quality bidirectional contextualization with sequence-level parallelism, as popularized by BERT-style architectures. Recently, Avey was introduced as an autoregressive, attention-free alternative that naturally admits an encoder-only adaptation. In this paper, we reformulate Avey for the encoder-only paradigm and propose several innovations to its architecture, including decoupled static and dynamic parameterizations, stability-oriented normalization, and neural compression. Results show that this reformulated architecture compares favorably to four widely used Transformer-based encoders, consistently outperforming them on standard token-classification and information-retrieval benchmarks while scaling more efficiently to long contexts.
Compact pretrained bidirectional encoders based on Avey architecture outperform Transformer-based models on token classification and information retrieval tasks while scaling more efficiently to long contexts.
bidirectional encoders self-attention BERT-style architectures autoregressive models attention-free alternative encoder-only adaptation decoupled parameterization stability-oriented normalization neural compression Transformer-based encoders token-classification information-retrieval