dleemiller/finecat-nli-v2-l

FineCat-NLI v2 Large

Try it: live demo Space

Natural Language Inference (NLI): the Swiss Army knife of encoders

Provide a text (premise) and statement (hypothesis), and it returns calibrated probabilities that the text entails, contradicts, or is neutral to the statement. A multiple-choice question is one statement per option.

Examples

Use	Text → statement	Result
Routing / triage	request → "The user wants to set an alarm."	97.7% (MASSIVE intents)
RAG grounding	retrieved passages → generated answer	0.81 F1 (RAGTruth)
Guardrails / Moderation	message → policy statement	92.3% (CrossingGuard), 94.3% (Wikipedia insults)
Zero-shot classification / tagging	review → "This review is positive."	92.3% (18 tasks)
Clinical extraction	clinical note → "The patient is unemployed."	98.4% (SDOH-NLI)

400M parameters · documents up to 8k tokens · ~1,200 pairs/s · Apache-2.0

from sentence_transformers import CrossEncoder

model = CrossEncoder("dleemiller/finecat-nli-v2-l")
text = "My card was charged twice for the same order."
labels = ["billing", "shipping", "account access"]
probs = model.predict([(text, f"This text is about {l}.") for l in labels], apply_softmax=True)
print({l: round(float(p[0]), 3) for l, p in zip(labels, probs)})  # p[0] = P(entailment)
# {'billing': 0.89, 'shipping': 0.12, 'account access': 0.02}

More in Recommended Applications and the 45-dataset evaluation.

---

Table of Contents

• NLI Evaluation Results
• Usage
• Recommended Applications
• Extended Evaluation (45 datasets)
• Training Details
• Acknowledgments
• Citation

---

NLI Evaluation Results

Comparison on the OpenNLI leaderboard (v0.2.0): every model scored with the same script at its own maximum context. Skill Index = the leaderboard score: Brier skill of the model's probabilities versus a uniform guess (0 = guessing, 100 = always right with full confidence), at the model's temperature fitted on the public calibration set, averaged over 4 categories (44 datasets). Accuracy = macro accuracy over the same datasets; the dataset columns are accuracy. CAL = calibration error as shipped (ECE %, lower is better). Throughput: OpenNLI speed test (MNLI pairs, batches of 128, tokenization excluded; Nvidia RTX PRO 6000 Blackwell Max-Q).

Model	Params	Skill Index	Accuracy	mnli	mnli_mismatched	snli	anli_r1	anli_r2	anli_r3	wanli	lingnli	CAL	Throughput (pairs/s)
dleemiller/finecat-nli-ultra-l	378M	58.6	0.8213	0.9127	0.9132	0.9302	0.7910	0.6400	0.5808	0.7698	0.8708	6.6	1,487
dleemiller/finecat-nli-v2-l	396M	57.8	0.8182	0.9164	0.9122	0.9329	0.7760	0.6040	0.5767	0.7664	0.8775	6.5	1,224
tasksource/ModernBERT-large-nli*	396M	37.7	0.7246	0.8981	0.9223	0.9192	0.7290	0.5130	0.4917	0.6970	0.8497	13.2	1,218
dleemiller/finecat-nli-l*	396M	37.5	0.7292	0.9155	0.9269	0.9162	0.7470	0.5680	0.5425	0.7708	0.8745	20.9	1,225
MoritzLaurer/DeBERTa-v3-large-mnli-fever-anli-ling-wanli	435M	27.9	0.6767	0.9123	0.9077	0.8897	0.7950	0.6820	0.6392	0.7696	0.8815	25.3	1,092
sileod/deberta-v3-large-tasksource-nli*	435M	24.8	0.6341	0.9298	0.9447	0.9211	0.7370	0.5770	0.4975	0.6786	0.8673	17.9	1,091
facebook/bart-large-mnli	407M	13.3	0.5827	0.9027	0.8998	0.8894	0.4220	0.3140	0.3258	0.6262	0.8473	28.7	1,708
cross-encoder/nli-deberta-v3-large	435M	13.1	0.5911	0.9018	0.9046	0.9221	0.5300	0.4180	0.3758	0.6548	0.8465	37.4	1,094

^* trained on one or more OpenNLI test sets; those scores are replaced by estimates, as on the leaderboard.

---

Usage

Label Map:

• entailment: 0
• neutral: 1
• contradiction: 2

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load the model and run inference.

from sentence_transformers import CrossEncoder
import numpy as np

model = CrossEncoder("dleemiller/finecat-nli-v2-l")
id2label = model.model.config.id2label  # {0:'entailment', 1:'neutral', 2:'contradiction'}

pairs = [
    ("The glass fell off the counter and shattered on the tile.",
     "The glass broke when it hit the floor."),          # E
    ("The store opens at 9 a.m. every day.",
     "The store opens at 7 a.m. on weekdays."),          # C
    ("A researcher presented results at the conference.",
     "The presentation won the best paper award."),       # N
    ("It started raining heavily, so the match was postponed.",
     "The game was delayed due to weather."),             # E
    ("Every seat on the flight was taken.",
     "There were several empty seats on the plane."),     # C
]

logits = model.predict(pairs)  # shape: (5, 3)

for (prem, hyp), row in zip(pairs, logits):
    pred_idx = int(np.argmax(row))
    pred = id2label[pred_idx]
    print(f"[{pred}]  Premise: {prem}  |  Hypothesis: {hyp}")

---

Recommended Applications

Frame each check as a premise (text) and hypothesis (statement), and use the entailment probability.

Application	Example hypothesis	Supporting results
Zero-shot tagging / topic classification	"This text is about billing."	zs_meta, fin_topics
Routing and decisions in agent pipelines	"The user wants to cancel an order."	zs_meta, mnli, snli
RAG grounding / hallucination checks	Each answer sentence vs. retrieved passages	ragtruth, wice, vitaminc
Contract and policy review	"The agreement permits sharing data with third parties."	contractnli_a/b, control, docnli
Biomedical and health claim checking	Claim vs. abstract or clinical note	sdoh_nli, biosift, bionli, scifact, pubhealth
Dialogue fact-checking	Assistant reply vs. knowledge source	dialfact, vitaminc
Guardrails / policy matching	Policy statement vs. user or model text	crossingguard
Stance detection	"The author supports the proposal."	vast

Less suited: financial and general legal verification (findver, legal_domain), and adversarial reasoning (anli_r3, nope_adv).

---

Extended Evaluation (45 datasets)

Accuracy on the test splits of 45 NLI and adjacent datasets. 2-way datasets (entailed vs. not) are scored as entailment vs. any non-entailment prediction. DocNLI and VitaminC are evaluated on fixed random subsets of 1,000 and 10,000 pairs. FRANK (news-summary factuality) is scored per summary sentence: each sentence vs. its source article, labelled by the majority annotator judgment.

Dataset	n	finecat-nli-v2-l	27B teacher (fast mode)
finecat	18012	0.8363	0.8656
mnli	9815	0.9165	0.9311
mnli_mismatched	9830	0.9123	0.9305
snli	9824	0.9329	0.9399
anli_r1	1000	0.7770	0.8630
anli_r2	1000	0.6030	0.7490
anli_r3	1200	0.5767	0.7100
wanli	5000	0.7668	0.7796
lingnli	7361	0.8773	0.8958
glue_diagnostic	1100	0.7582	0.8118
med	5380	0.7935	0.8039
nan_nli	258	0.7054	0.7403
scone_test	1000	1.0000	0.9980
nmonli_test	200	1.0000	1.0000
conjnli	623	0.7223	0.7335
nope_adv	345	0.5594	0.5768
nli4ct	199	0.7688	0.8693
sdoh_nli	990	0.9838	0.9848
healthver	230	0.7696	0.7565
scifact	300	0.8600	0.8433
biosift	1000	0.9200	0.9210
scientsbank	971	0.7477	0.7683
beetle	819	0.6447	0.6325
legal_domain	1163	0.5211	0.6389
findver	500	0.6640	0.7660
dialfact	1499	0.8432	0.8586
frank_sent	3176	0.7317	0.8234
ragtruth	1497	0.8417	0.8691
vast	1500	0.7860	0.8053
zs_meta	5400	0.9231	0.9344
seahorse	800	0.8450	0.8638
pubhealth	600	0.8517	0.8650
bionli	600	0.8917	0.9333
fin_topics	600	0.9667	0.9633
vitaminc	10000	0.8819	0.9351
docnli	1000	0.8350	—
contractnli_a	1707	0.9268	0.9285
contractnli_b	2074	0.8925	0.9021
control	800	0.7100	0.8087
fol	10277	0.8398	0.8430
scitail	2126	0.9624	0.9652
wice	958	0.8653	0.9144
sick	4906	0.9240	0.9213
crossingguard	5397	0.9227	0.9344
hans	15000	0.8578	0.9209

Macro average over the 45 datasets: 0.8204.

Training Details

This model is a fine-tune of jhu-clsp/ettin-encoder-400m, trained on the dleemiller/FineCat-NLI-Reason dataset, a compilation of several high-quality NLI data sources with quality screening and reduction of easy samples in the training split. Also includes additional NLI, fact verification and long document sources (2.05M pairs in total). The training uses logit distillation from an NLI finetune of Qwen3.8-27B.

Distillation loss looks like this: $$\begin{array}{cccc}& {\displaystyle \mathcal{L}=\alpha \cdot {\mathcal{L}}_{\text{CE}}(z^{(s)},y)+\beta \cdot T^2\cdot \mathrm{K}\mathrm{L}(\sigma (z^{(t)}\mathrm{/}T)\mathrm{\parallel }\sigma (z^{(s)}\mathrm{/}T))}& & \end{array}\backslash begin\{equation\}\; \backslash mathcal\{L\}\; =\; \backslash alpha\; \backslash cdot\; \backslash mathcal\{L\}\_\{\backslash text\{CE\}\}(z^\{(s)\},\; y)\; +\; \backslash beta\; \backslash cdot\; T^2\; \backslash cdot\; \backslash mathrm\{KL\}\backslash left(\backslash sigma(z^\{(t)\}/T)\; \backslash ,\backslash |\backslash ,\; \backslash sigma(z^\{(s)\}/T)\backslash right)\; \backslash end\{equation\}$$

where $z^{(s)}z^\{(s)\}$ and $z^{(t)}z^\{(t)\}$ are the student and teacher logits, $yy$ are the ground truth labels, $\alpha =0.2\backslash alpha\; =\; 0.2$, $\beta =0.8\backslash beta\; =\; 0.8$ and $T=1T\; =\; 1$.

Training inputs keep the full documents the teacher labelled (up to 8k tokens). The model was trained for 3 epochs at a constant learning rate followed by a 1-epoch cooldown. The released weights are an average of the final weights and earlier exponential-moving-average checkpoints, with the output temperature calibrated on the validation split.

---

Acknowledgments

We thank the creators and contributors of jhu-clsp (Ettin) and the Qwen team for making their work available. This model would not be possible without their efforts and open source contributions.

Citation

@misc{nli-compiled-2025,
  title = {FineCat NLI Dataset},
  author = {Lee Miller},
  year = {2025},
  howpublished = {Refined compilation of 6 major NLI datasets}
}