Note: This document is available both in english and portuguese. The portuguese version can be found bellow the english one

NorBERTo

NorBERTo is a pretrained ModernBERT encoder for Brazilian Portuguese that achieves state-of-the-art performance on Portuguese NLP benchmarks, including PLUE and ASSIN 2 textual entailment. It is available in two sizes: Base and Large.

NorBERTo is trained on Aurora-PT, a 331-billion-token Brazilian Portuguese corpus developed by Itaú Unibanco and ICTi. Compared to previous Portuguese encoders, NorBERTo features long-context support (up to 8,192 tokens), efficient local–global attention (RoPE), and a vocabulary trained on Portuguese data.

For further information, please refer to the paper or the Aurora-PT dataset.

Available models

|:------|:------------:|:-------:|:-------:|

| Itau-Unibanco/NorBERTo-base | ModernBERT-Base | 22 | ~150M |

| Itau-Unibanco/NorBERTo-large | ModernBERT-Large | 28 | ~395M |

Usage


from transformers import AutoTokenizer, AutoModel

 

model = AutoModel.from_pretrained('Itau-Unibanco/NorBERTo-base')

tokenizer = AutoTokenizer.from_pretrained('Itau-Unibanco/NorBERTo-base')

Masked language modeling


from transformers import pipeline

 

pipe = pipeline('fill-mask', model='Itau-Unibanco/NorBERTo-base')

 

pipe('O Brasil é um país de dimensões [MASK].')

For embeddings


import torch

 

model = AutoModel.from_pretrained('Itau-Unibanco/NorBERTo-base')

tokenizer = AutoTokenizer.from_pretrained('Itau-Unibanco/NorBERTo-base')

 

input_ids = tokenizer.encode('O Brasil é um país de dimensões continentais.', return_tensors='pt')

 

with torch.no_grad():

    outs = model(input_ids)

    encoded = outs[0][0, 1:-1]  # Ignore [CLS] and [SEP] special tokens

Results

ASSIN 2

|:------|:----:|:-------------:|:--------------------:|

| Albertina PT-BR | base | 0.874 | 0.826 |

| BERTimbau | base | 0.883 | 0.836 |

| BERTimbau | large | 0.889 | 0.852 |

| mmBERT | base | 0.896 | 0.821 |

| NorBERTo | base | 0.890 | 0.736 |

| NorBERTo (cross encoder) | large | 0.903 | 0.766 |

| NorBERTo (sequence classification) | large | 0.904 | — |

PLUE

|:------|:----:|:----:|:---:|:----:|

| Albertina PT-BR | base | 0.878 | 0.646 | 0.549 |

| BERTimbau | large | 0.887 | 0.755 | 0.563 |

| mmBERT | base | 0.905 | 0.758 | 0.563 |

| NorBERTo | base | 0.893 | 0.722 | 0.577 |

| NorBERTo | large | 0.919 | 0.769 | 0.577 |

Citation

If you use our work, please cite:


@inproceedings{silva-etal-2026-norberto,
    title = "{N}or{BERT}o: A {M}odern{BERT} Model Trained for {P}ortuguese with 331 Billion Tokens Corpus",
    author = "Silva, Enzo S. N.  and
      Costa, Pablo B.  and
      Vlasman, Raphael C.  and
      Costa, Rosimeire P.  and
      Silva, Henrique L. P.  and
      Pellicer, Lucas F. A. O.  and
      Rinaldo, Guilherme  and
      Almeida, Renato A.  and
      Rabbani, Darian S. R.  and
      Oestreich, Cinthya O.  and
      Carid{\'a}, Vinicius F.",
    editor = "Souza, Marlo  and
      de-Dios-Flores, Iria  and
      Santos, Diana  and
      Freitas, Larissa  and
      Souza, Jackson Wilke da Cruz  and
      Ribeiro, Eug{\'e}nio",
    booktitle = "Proceedings of the 17th International Conference on Computational Processing of {P}ortuguese ({PROPOR} 2026) - Vol. 1",
    month = apr,
    year = "2026",
    address = "Salvador, Brazil",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2026.propor-1.18/",
    pages = "183--193",
    ISBN = "979-8-89176-387-6",
    abstract = "High-quality corpora are essential for advancing Natural Language Processing (NLP) in Portuguese. Building on previous encoder-only models such as BERTimbau and Albertina PT-BR, we introduce NorBERTo, a modern encoder based on the ModernBERT architecture, featuring long-context support and efficient attention mechanisms. NorBERTo is trained on Aurora-PT, a newly curated Brazilian Portuguese corpus comprising 331 billion GPT-2 tokens collected from diverse web sources and existing multilingual datasets. We systematically benchmark NorBERTo against strong baselines on semantic similarity, textual entailment and classification tasks using standardized datasets such as ASSIN 2 and PLUE. On PLUE, NorBERTo-large achieves the best results among the encoder models we evaluated, notably reaching 0.9191 F1 on MRPC and 0.7689 accuracy on RTE. On ASSIN 2, NorBERTo-large attains the highest entailment F1 (0.904) among all encoders considered, while remaining competitive, but Albertina-900M and BERTimbau-large still hold an advantage. To the best of our knowledge, Aurora-PT is currently the largest openly available monolingual Portuguese corpus, surpassing previous resources. NorBERTo provides a modern, mid-sized encoder designed for realistic deployment scenarios: it is straightforward to fine-tune, efficient to serve, and well suited as a backbone for retrieval-augmented generation and other downstream Portuguese NLP systems."
}

Português

NorBERTo

NorBERTo é um encoder ModernBERT pré-treinado para o português brasileiro que alcança resultados estado da arte em benchmarks de PLN em português, incluindo PLUE e inferência textual no ASSIN 2. Está disponível em dois tamanhos: Base e Large.

O NorBERTo foi treinado no Aurora-PT, um corpus de 331 bilhões de tokens em português brasileiro desenvolvido pelo Itaú Unibanco e pelo ICTi. Em comparação com encoders anteriores para o português, o NorBERTo oferece suporte a contextos longos (até 8.192 tokens), atenção local–global eficiente (RoPE) e vocabulário treinado em dados em português.

Para mais informações, consulte o artigo ou o dataset Aurora-PT.

Modelos disponíveis

|:-------|:-----------:|:--------:|:-----------:|

| Itau-Unibanco/NorBERTo-base | ModernBERT-Base | 22 | ~150M |

| Itau-Unibanco/NorBERTo-large | ModernBERT-Large | 28 | ~395M |

Como usar


from transformers import AutoTokenizer, AutoModel

 

model = AutoModel.from_pretrained('Itau-Unibanco/NorBERTo-base')

tokenizer = AutoTokenizer.from_pretrained('Itau-Unibanco/NorBERTo-base')

Masked language modeling


from transformers import pipeline

 

pipe = pipeline('fill-mask', model='Itau-Unibanco/NorBERTo-base')

 

pipe('O Brasil é um país de dimensões [MASK].')

Para embeddings


import torch

 

model = AutoModel.from_pretrained('Itau-Unibanco/NorBERTo-base')

tokenizer = AutoTokenizer.from_pretrained('Itau-Unibanco/NorBERTo-base')

 

input_ids = tokenizer.encode('O Brasil é um país de dimensões continentais.', return_tensors='pt')

 

with torch.no_grad():

    outs = model(input_ids)

    encoded = outs[0][0, 1:-1]  # Ignore [CLS] e [SEP]

Resultados

ASSIN 2

|:-------|:-------:|:-------------:|:----------------------:|

| Albertina PT-BR | base | 0,874 | 0,826 |

| BERTimbau | base | 0,883 | 0,836 |

| BERTimbau | large | 0,889 | 0,852 |

| mmBERT | base | 0,896 | 0,821 |

| NorBERTo | base | 0,890 | 0,736 |

| NorBERTo (cross encoder) | large | 0,903 | 0,766 |

| NorBERTo (sequence classification) | large | 0,904 | — |

PLUE

|:-------|:-------:|:----:|:---:|:----:|

| Albertina PT-BR | base | 0,878 | 0,646 | 0,549 |

| BERTimbau | large | 0,887 | 0,755 | 0,563 |

| mmBERT | base | 0,905 | 0,758 | 0,563 |

| NorBERTo | base | 0,893 | 0,722 | 0,577 |

| NorBERTo | large | 0,919 | 0,769 | 0,577 |

Citação

Se utilizar este trabalho, por favor cite:


@inproceedings{silva-etal-2026-norberto,
    title = "{N}or{BERT}o: A {M}odern{BERT} Model Trained for {P}ortuguese with 331 Billion Tokens Corpus",
    author = "Silva, Enzo S. N.  and
      Costa, Pablo B.  and
      Vlasman, Raphael C.  and
      Costa, Rosimeire P.  and
      Silva, Henrique L. P.  and
      Pellicer, Lucas F. A. O.  and
      Rinaldo, Guilherme  and
      Almeida, Renato A.  and
      Rabbani, Darian S. R.  and
      Oestreich, Cinthya O.  and
      Carid{\'a}, Vinicius F.",
    editor = "Souza, Marlo  and
      de-Dios-Flores, Iria  and
      Santos, Diana  and
      Freitas, Larissa  and
      Souza, Jackson Wilke da Cruz  and
      Ribeiro, Eug{\'e}nio",
    booktitle = "Proceedings of the 17th International Conference on Computational Processing of {P}ortuguese ({PROPOR} 2026) - Vol. 1",
    month = apr,
    year = "2026",
    address = "Salvador, Brazil",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2026.propor-1.18/",
    pages = "183--193",
    ISBN = "979-8-89176-387-6",
    abstract = "High-quality corpora are essential for advancing Natural Language Processing (NLP) in Portuguese. Building on previous encoder-only models such as BERTimbau and Albertina PT-BR, we introduce NorBERTo, a modern encoder based on the ModernBERT architecture, featuring long-context support and efficient attention mechanisms. NorBERTo is trained on Aurora-PT, a newly curated Brazilian Portuguese corpus comprising 331 billion GPT-2 tokens collected from diverse web sources and existing multilingual datasets. We systematically benchmark NorBERTo against strong baselines on semantic similarity, textual entailment and classification tasks using standardized datasets such as ASSIN 2 and PLUE. On PLUE, NorBERTo-large achieves the best results among the encoder models we evaluated, notably reaching 0.9191 F1 on MRPC and 0.7689 accuracy on RTE. On ASSIN 2, NorBERTo-large attains the highest entailment F1 (0.904) among all encoders considered, while remaining competitive, but Albertina-900M and BERTimbau-large still hold an advantage. To the best of our knowledge, Aurora-PT is currently the largest openly available monolingual Portuguese corpus, surpassing previous resources. NorBERTo provides a modern, mid-sized encoder designed for realistic deployment scenarios: it is straightforward to fine-tune, efficient to serve, and well suited as a backbone for retrieval-augmented generation and other downstream Portuguese NLP systems."
}

Downloads last month: 4,804

Safetensors

Model size

0.1B params

Tensor type

F32

Model tree for Itau-Unibanco/NorBERTo-base

Quantizations

1 model

Paper for Itau-Unibanco/NorBERTo-base

NorBERTo: A ModernBERT Model Trained for Portuguese with 331 Billion Tokens Corpus

Paper • 2605.00086 • Published Apr 30 • 1