| --- |
| language: |
| - tr |
| license: apache-2.0 |
| library_name: sentence-transformers |
| base_model: ytu-ce-cosmos/modernbert-tr-base |
| pipeline_tag: sentence-similarity |
| tags: |
| - sentence-transformers |
| - sentence-similarity |
| - feature-extraction |
| - mteb |
| - onnx |
| - encoderfile |
| model-index: |
| - name: modernbert-tr-embed |
| results: |
| - task: |
| type: Retrieval |
| name: ArguAnaTR |
| dataset: |
| type: trmteb/arguana-tr |
| name: MTEB ArguAnaTR |
| config: default |
| split: test |
| revision: main |
| metrics: |
| - type: ndcg_at_10 |
| value: 50.01 |
| - task: |
| type: Retrieval |
| name: CQADupstackGamingRetrievalTR |
| dataset: |
| type: trmteb/cqadupstack-gaming-tr |
| name: MTEB CQADupstackGamingRetrievalTR |
| config: default |
| split: test |
| revision: main |
| metrics: |
| - type: ndcg_at_10 |
| value: 56.37 |
| - task: |
| type: Retrieval |
| name: FiQA2018TR |
| dataset: |
| type: trmteb/fiqa-tr |
| name: MTEB FiQA2018TR |
| config: default |
| split: test |
| revision: main |
| metrics: |
| - type: ndcg_at_10 |
| value: 46.2 |
| - task: |
| type: Retrieval |
| name: MSMarcoTRRetrieval |
| dataset: |
| type: trmteb/msmarco-tr |
| name: MTEB MSMarcoTRRetrieval |
| config: default |
| split: test |
| revision: main |
| metrics: |
| - type: ndcg_at_10 |
| value: 57.87 |
| - task: |
| type: PairClassification |
| name: MnliTr |
| dataset: |
| type: trmteb/multinli_tr |
| name: MTEB MnliTr |
| config: default |
| split: test_matched |
| revision: main |
| metrics: |
| - type: max_ap |
| value: 65.92 |
| - task: |
| type: Retrieval |
| name: NFCorpusTR |
| dataset: |
| type: trmteb/nfcorpus-tr |
| name: MTEB NFCorpusTR |
| config: default |
| split: test |
| revision: main |
| metrics: |
| - type: ndcg_at_10 |
| value: 9.47 |
| - task: |
| type: Retrieval |
| name: QuoraRetrievalTR |
| dataset: |
| type: trmteb/quora-tr |
| name: MTEB QuoraRetrievalTR |
| config: default |
| split: test |
| revision: main |
| metrics: |
| - type: ndcg_at_10 |
| value: 95.18 |
| - task: |
| type: Retrieval |
| name: SCIDOCSTR |
| dataset: |
| type: trmteb/scidocs-tr |
| name: MTEB SCIDOCSTR |
| config: default |
| split: test |
| revision: main |
| metrics: |
| - type: ndcg_at_10 |
| value: 3.37 |
| - task: |
| type: STS |
| name: STSbTR |
| dataset: |
| type: trmteb/stsb-tr |
| name: MTEB STSbTR |
| config: default |
| split: test |
| revision: main |
| metrics: |
| - type: cosine_spearman |
| value: 77.62 |
| - task: |
| type: Retrieval |
| name: SciFactTR |
| dataset: |
| type: trmteb/scifact-tr |
| name: MTEB SciFactTR |
| config: default |
| split: test |
| revision: main |
| metrics: |
| - type: ndcg_at_10 |
| value: 77.06 |
| - task: |
| type: PairClassification |
| name: SnliTr |
| dataset: |
| type: trmteb/snli_tr |
| name: MTEB SnliTr |
| config: default |
| split: test |
| revision: main |
| metrics: |
| - type: max_ap |
| value: 67.29 |
| - task: |
| type: Retrieval |
| name: SquadTRRetrieval |
| dataset: |
| type: trmteb/squad-tr |
| name: MTEB SquadTRRetrieval |
| config: default |
| split: test |
| revision: main |
| metrics: |
| - type: ndcg_at_10 |
| value: 75.93 |
| - task: |
| type: Classification |
| name: THYSentimentClassification |
| dataset: |
| type: trmteb/thy_sa |
| name: MTEB THYSentimentClassification |
| config: default |
| split: test |
| revision: main |
| metrics: |
| - type: accuracy |
| value: 68.14 |
| - task: |
| type: Retrieval |
| name: TQuadRetrieval |
| dataset: |
| type: trmteb/tquad |
| name: MTEB TQuadRetrieval |
| config: default |
| split: test |
| revision: main |
| metrics: |
| - type: ndcg_at_10 |
| value: 87.14 |
| - task: |
| type: Classification |
| name: TSTimelineNewsCategoryClassification |
| dataset: |
| type: trmteb/ts_timeline_news_category |
| name: MTEB TSTimelineNewsCategoryClassification |
| config: default |
| split: test |
| revision: main |
| metrics: |
| - type: accuracy |
| value: 67.91 |
| - task: |
| type: Classification |
| name: Turkish75NewsClassification |
| dataset: |
| type: trmteb/75haber |
| name: MTEB Turkish75NewsClassification |
| config: default |
| split: test |
| revision: main |
| metrics: |
| - type: accuracy |
| value: 93.33 |
| - task: |
| type: Clustering |
| name: TurkishAbstractCorpusClustering |
| dataset: |
| type: trmteb/ts_abstract_corpus_p2p |
| name: MTEB TurkishAbstractCorpusClustering |
| config: default |
| split: test |
| revision: main |
| metrics: |
| - type: v_measure |
| value: 62.22 |
| - task: |
| type: Clustering |
| name: TurkishColumnWritingClustering |
| dataset: |
| type: trmteb/630koseyazisi_p2p |
| name: MTEB TurkishColumnWritingClustering |
| config: default |
| split: test |
| revision: main |
| metrics: |
| - type: v_measure |
| value: 64.42 |
| - task: |
| type: Classification |
| name: TurkishIronyClassification |
| dataset: |
| type: trmteb/irony-tr |
| name: MTEB TurkishIronyClassification |
| config: default |
| split: test |
| revision: main |
| metrics: |
| - type: accuracy |
| value: 57.67 |
| - task: |
| type: Classification |
| name: TurkishMovieSentimentClassification |
| dataset: |
| type: asparius/Turkish-Movie-Review |
| name: MTEB TurkishMovieSentimentClassification |
| config: default |
| split: test |
| revision: 409a4415cce5f6bcfca6d5f3ca3c408211ca00b3 |
| metrics: |
| - type: accuracy |
| value: 89.34 |
| - task: |
| type: Classification |
| name: TurkishNewsCategoryClassification |
| dataset: |
| type: trmteb/news-cat |
| name: MTEB TurkishNewsCategoryClassification |
| config: default |
| split: test |
| revision: main |
| metrics: |
| - type: accuracy |
| value: 93.64 |
| - task: |
| type: Classification |
| name: TurkishOffensiveLanguageClassification |
| dataset: |
| type: trmteb/offenseval |
| name: MTEB TurkishOffensiveLanguageClassification |
| config: default |
| split: test |
| revision: main |
| metrics: |
| - type: accuracy |
| value: 71.51 |
| - task: |
| type: Classification |
| name: TurkishProductSentimentClassification |
| dataset: |
| type: asparius/Turkish-Product-Review |
| name: MTEB TurkishProductSentimentClassification |
| config: default |
| split: test |
| revision: ad861e463abda351ff65ca5ac0cc5985afe9eb99 |
| metrics: |
| - type: accuracy |
| value: 70.51 |
| - task: |
| type: BitextMining |
| name: WMT16BitextMining |
| dataset: |
| type: trmteb/wmt16_en_tr |
| name: MTEB WMT16BitextMining |
| config: default |
| split: test |
| revision: main |
| metrics: |
| - type: f1 |
| value: 94.09 |
| - task: |
| type: PairClassification |
| name: XNLI |
| dataset: |
| type: mteb/xnli |
| name: MTEB XNLI |
| config: default |
| split: test |
| revision: 09698e0180d87dc247ca447d3a1248b931ac0cdb |
| metrics: |
| - type: max_ap |
| value: 74.44 |
| - task: |
| type: Retrieval |
| name: XQuADRetrieval |
| dataset: |
| type: google/xquad |
| name: MTEB XQuADRetrieval |
| config: default |
| split: validation |
| revision: 51adfef1c1287aab1d2d91b5bead9bcfb9c68583 |
| metrics: |
| - type: ndcg_at_10 |
| value: 95.03 |
| --- |
| |
|
|
| <p align="center"> |
| <img src="assets/logo.webp" width="20%" alt="ModernBERT-TR Embed" /> |
| </p> |
| <h1 align="center">ModernBERT-TR Embed</h1> |
|
|
| A 150M-parameter Turkish text-embedding model. |
|
|
| - Base model: [`ytu-ce-cosmos/modernbert-tr-base`](https://huggingface.co/ytu-ce-cosmos/modernbert-tr-base). |
| - Distilled from `Qwen/Qwen3-Embedding-8B`. |
|
|
| ## Results |
|
|
| | Model | Params | Retr | Classif | PairCls | Cluster | STS | Bitext | **Mean** | |
| |---|---|---|---|---|---|---|---|---| |
| | **ModernBERT-TR-Embed (ours)** | 150M | 59.4 | **76.5** | **69.2** | **63.3** | 77.6 | 94.1 | **68.14** | |
| | ytu-ce-cosmos/turkish-e5-large | 560M | 61.5 | 72.6 | 62.8 | 60.9 | 80.0 | **99.3** | 67.17 | |
| | microsoft/harrier-oss-v1-0.6b | 600M | 60.1 | 71.1 | 58.6 | 63.3 | 74.5 | 98.6 | 65.57 | |
| | intfloat/multilingual-e5-large | 560M | 61.7 | 69.2 | 65.6 | 60.8 | **81.0** | 99.0 | 66.56 | |
| | Qwen/Qwen3-Embedding-4B | 4B | **63.1** | 70.2 | 60.1 | 61.3 | 77.0 | 97.8 | 66.69 | |
|
|
| ## How was this model trained? |
|
|
| 1. We embedded ~7.9M Turkish passages with the teacher, then trained our model to reproduce those embeddings. A projector maps the teacher's 4096-d vectors down to our model's 768 dimensions. Following [Jasper/Stella distillation recipe](https://arxiv.org/abs/2412.19048), a three-term loss aligns the embeddings from both: a per-passage cosine loss, a similarity-matrix loss matching the student's and teacher's Gram matrices within batch, and a CoSENT-style hinge that reproduces the teacher's pairwise-similarity ordering. |
|
|
| 2. From the distilled model we ran three fine-tunings, all supervised by teacher embeddings: |
| - Retrieval: the student ranks the correct passage above hard negatives for a given query. Trained with an [InfoNCE](https://arxiv.org/abs/1807.03748) contrastive loss over in-batch and hard negatives, plus a KL term matching the teacher's softmax ranking over each query's candidates. |
| - Multi-task: the retrieval objective plus Turkish language-understanding tasks: NLI, STS as in [CoSENT](https://kexue.fm/archives/8847) on teacher cosine, supervised contrastive classification as in [SupCon](https://arxiv.org/abs/2004.11362), and a replay of the cosine distillation on classification text. |
| - Cross-lingual: the multi-task recipe with rebalanced task weights and added English retrieval passages, to improve English-Turkish alignment. |
|
|
| 3. We weight-average the checkpoints described above into a single model. |
|
|
|
|
| ## Usage |
|
|
| ### sentence-transformers |
|
|
| ```python |
| from sentence_transformers import SentenceTransformer |
| model = SentenceTransformer("mrbesher/modernbert-tr-embed") |
| |
| emb = model.encode(["bir cümle", "başka bir cümle"], normalize_embeddings=True) |
| |
| q = model.encode(["soru"], prompt_name="query", normalize_embeddings=True) |
| d = model.encode(["döküman"], normalize_embeddings=True) |
| ``` |
|
|
| The retrieval (see `config_sentence_transformers.json`): |
| `Instruct: Given a web search query, retrieve relevant passages that answer the query\nQuery:{text}`. |
|
|
| ### ONNX Runtime |
|
|
| The `onnx/` folder has the grapgh for the token embeddings, it includes 3 graphs: token embeddings, mean-pool and L2-norm. |
|
|
| ```python |
| from sentence_transformers import SentenceTransformer |
| model = SentenceTransformer("mrbesher/modernbert-tr-embed", backend="onnx", |
| model_kwargs={"file_name": "onnx/model_fp16.onnx"}) |
| ``` |
|
|
| ### Text Embeddings Inference (TEI) |
|
|
| TEI is L2-normalizes by default. Pass per-request prompts for retrieval queries. |
|
|
| ```bash |
| text-embeddings-router --model-id mrbesher/modernbert-tr-embed --dtype float16 |
| ``` |
|
|
| ### Encoderfile |
|
|
| See the companion [encoderfile](https://huggingface.co/ytu-ce-cosmos/modernbert-tr-embed-encoderfile) repo. |
|
|
| ## Training data |
|
|
| Turkish retrieval (msmarco-tr, Squad-TR train/dev), Turkish NLI (`boun-tabi/nli_tr` train), Turkish STS-B (train), and Turkish classification-domain text (product reviews, news, social), all teacher-supervised. We check for leaks with text-hash against every MTEB(Turkish) test split. |
|
|
| ## License & attribution |
|
|
| - License: `apache-2.0`. |
