Instructions to use ba-research/Qev1.0-Qwen3.5-4B with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- PEFT
How to use ba-research/Qev1.0-Qwen3.5-4B with PEFT:
Task type is invalid.
- Notebooks
- Google Colab
- Kaggle
Qev1.0-Qwen3.5-4B
Qev 1.0 4B is an experimental decision model built on Qwen3.5-4B. It scores candidate answers to Choice, Noul and Score questions. This repository contains a LoRA adapter, a separate learned decision head and calibration metadata; it requires the base model and the Open-Jev decision-model implementation. It is not a standalone chat model.
Intended use and limitations
Use this checkpoint for research and evaluation of structured decisions. Test it on representative tasks before using it in a router or consequential workflow. The development panel is provisional and not frozen. Independent generalization, reliable calibration across domains, cost savings and parity with other decision models have not been established. Outputs can be wrong; do not treat a score as a guarantee.
Evaluation
On the 2,352-item OpenDecisions validation set used during development, this checkpoint obtained a capability score of 65.98, with Choice, Noul and Score accuracies of 68.25%, 77.30% and 56.46%, respectively. These are development-set results, not a fresh independent test. The source benchmark and scoring definitions are described in OpenDecisions. Results may be affected by model-selection and data-overlap limitations.
4B all-around OpenDecisions profile (local, 13 November 2026)
Both pinned checkpoints were run on the same 2,352 OpenDecisions 1.0.1 items (dataset SHA-256 78614972006c5e5efbbfc121ce31e15cf576ef6c0cdf24a229b33d34d2bfcda6). This is public development data, not independent generalization evidence. Jeff's model card states that OpenDecisions informed training-length selection. The model-specific native prompts and inference paths differ, so these results do not isolate the effect of training.
| Metric | Qev 1.0 4B recheck | Jeff 1.1 Qwen 4B | Observed leader |
|---|---|---|---|
| Capability / Intelligence / Calibration | 66.09 / 44.70 / 87.49 | 79.35 / 68.46 / 90.23 | Jeff / Jeff / Jeff |
| Choice / Noul / Score accuracy | 68.34% / 77.13% / 56.46% | 83.87% / 88.40% / 73.81% | Jeff / Jeff / Jeff |
| Choice / Noul / Score competence | 59.23 / 49.39 / 25.48 | 79.57 / 74.55 / 51.25 | Jeff / Jeff / Jeff |
| Choice / Noul / Score calibration | 80.70 / 91.71 / 90.05 | 88.67 / 92.40 / 89.63 | Jeff / Jeff / Qev |
| Calibration-target TVD, 129 items (lower better) | 0.18760 | 0.08092 | Jeff |
| Hard-label NLL / multiclass Brier (lower better) | 0.79124 / 0.43067 | 0.48293 / 0.25889 | Jeff / Jeff |
| Top-label ECE, 10 bins (lower better) | 0.02177 | 0.03816 | Qev, point estimate |
| Coverage / refusals | 100% / 0 | 100% / 0 | Tie |
Paired family-cluster bootstrap (5,000 draws, 11 families): Qev minus Jeff Capability β13.25 points (95% CI β18.96 to β10.44), Intelligence β23.76 (β31.83 to β19.46), and Calibration β2.75 (β9.47 to +1.22). Accuracy differences were Choice β15.53 pp (β23.67 to β11.57), Noul β11.26 pp (β18.13 to β6.58), and Score β17.35 pp (β26.06 to β12.47). Supplementary paired differences were NLL +0.308 (+0.255 to +0.403), Brier +0.172 (+0.145 to +0.222), ECE β0.016 (β0.030 to +0.020; includes zero), and TVD +0.107 (+0.106 to +0.107). The TVD interval is conditional on bootstrap samples containing calibration-target items.
Latency and GPU memory were measured on the same RTX 5070 Ti but through different native serving protocols; they are descriptive, not controlled efficiency comparisons. The full method, model-specific hashes, row-level outputs and all intervals are linked in the Qev comparison report. Qev leads two calibration point estimates, while Jeff leads most measured quality metrics. Neither this profile nor a single Capability score establishes overall SOTA or generalization.
Training and provenance
The hybrid-v2 checkpoint used 1,000 selected decisions (400 Choice, 300 Noul, 300 Score), 250 optimizer steps, rank-8 LoRA and a separate 128-row calibration split. Training input SHA-256: a7f4c474442e8f577765630649e893324cd09257420c3a3b11f6c98d28939c25. The base revision is 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a.
The selected mixture includes 100 CommonsenseQA Choice examples, 100 Banking77 Choice controls, 150 Measuring Hate Speech Noul examples and 150 HelpSteer2 Score examples; the remainder includes Open-Jev-derived controls. The selection is not independently adjudicated or frozen. The HelpSteer2 dataset card lists CC-BY-4.0. The selected prompts' individual upstream origins have not been resolved. The upstream Qwen base card lists Apache-2.0. These facts do not independently establish the license of the adapter or all training-data permissions.
Files and loading
adapter/adapter_model.safetensorsandadapter/adapter_config.json: LoRA adapter.head.pt: separate learned decision head.temperature.json: calibration temperature; applied separately by the evaluation code.model.json: model and training configuration.SHA256SUMS.json: hashes of the staged model files.
The checked loader comes from Open-Jev commit bd4118882f733574a3250a4b65fe4d884130c08b. Its source is MIT licensed. Install that revision and its training dependencies, then point the loader at the downloaded checkpoint directory:
git clone https://github.com/Zefan-Cai/Open-Jev.git
cd Open-Jev
git checkout bd4118882f733574a3250a4b65fe4d884130c08b
pip install -e '.[train]'
from pathlib import Path
from jev.model import DecisionModel
checkpoint = Path("/path/to/downloaded/qev-checkpoint")
model = DecisionModel.load(checkpoint, device="cpu")
DecisionModel.load loads the base model, adapter and head; it does not apply temperature.json automatically. A local CPU load check succeeded with an existing local base copy and the staged adapter and head. A fresh Hub download, base-file hashes, clean-environment setup and inference quality were not verified by that check.
A local CPU smoke used a synthetic yes/no question and returned finite uncalibrated logits approximately [0.0, 1.8188] with the local base copy. This single observation is not an accuracy, calibration or clean-download test.
Qev 1.0 comparison by model size
Qev 1.0: comparison with same-size decision models
Snapshot: 13 November 2026. Qev's OpenDecisions results use a 2,352-item development set, not a fresh independent test. Capability averages the benchmark's intelligence and calibration scores; it is not accuracy. Public-231 accuracy is a separate local diagnostic detailed below. Unless marked local, competitor scores are reported by their authors or the benchmark authors using their own serving configurations. Tables list identified comparators, not a complete ranking or proof of global state of the art. Different benchmarks cannot be subtracted.
0.8B
| Model | Base | OpenDecisions Capability | Public 231 accuracy | Typed-decisions accuracy | Comparison to Qev 1.0 0.8B |
|---|---|---|---|---|---|
| Qev 1.0 0.8B | Qwen3.5-0.8B | 46.85 | 56.28% | 43.50%β | Our development result |
| Experimental local 0.8B pilot | Qwen3.5-0.8B | 50.45 | 56.28% | 58.15%β | Unreleased workflow-trained pilot; preliminary, not independently accepted |
| Experimental Tasksource 0.8B | Qwen3.5-0.8B | 49.32 | 57.58% | β | Unreleased broad-source checkpoint; probability quality and robustness tradeoffs |
| Kev 0.8B | Qwen3.5-0.8B-Base | 59.14 | 63.20% | 44.15%β | Local diagnostics: 12.29 Capability points and 6.93 public-231 accuracy points above Qev 1.0 |
| StartLux 1.1 0.8B Instant | Qwen3.5-0.8B | Not verified | β | β | Gap unknown; same-suite run needed |
| AA Decision 0.8B | Qwen3.5-0.8B | Not verified | β | β | Gap unknown; card reports different benchmarks |
| labqoat QEV 0.8B | Qwen3.5-0.8B | Not verified | β | β | Different project; gap unknown |
2B
| Model | Base | OpenDecisions Capability | Public 231 accuracy | Typed-decisions accuracy | Comparison to Qev 1.0 2B |
|---|---|---|---|---|---|
| Qev 1.0 2B | Qwen3.5-2B | 55.23 | 63.64% | 56.10%β | Our development result |
| Experimental local 2B pilot | Qwen3.5-2B | 51.93 | 70.13% | 69.40%β | Unreleased workflow-trained pilot; OpenDecisions regressed |
| Valen Decision 1.5 2B | Qwen3.5-2B | 62.12 | 81.39% | 69.15%β | Local runs: 6.90 Capability points, 17.75 public-231 and 13.05 typed-decisions accuracy points above Qev 2B |
| Decider 2B | Qwen3.5-2B-Base | Not verified | 77.49% | β | Local calibrated public-231 run; 13.85 points above Qev 1.0 |
| StartLux 1.1 2B Instant | Qwen3.5-2B | Not verified | β | β | Gap unknown; same-suite run needed |
| joey-1 | Qwen3.5-2B | Not verified | β | β | Gap unknown; same-suite run needed |
| Jev-Style 2B Decision v3 | Qwen3.5-2B | Not verified | β | β | Gap unknown; same-suite run needed |
4B
| Model | Base | OpenDecisions Capability | Public 231 accuracy | Typed-decisions accuracy | Comparison to Qev 1.0 4B |
|---|---|---|---|---|---|
| StartLux 1.1 4B Instant | Qwen3.5-4B | 82.2 | β | β | 16.22 points above Qev; 19.73% relative gap |
| Kev 4B | Qwen3.5-4B-Base | Not verified | 74.46% | 65.70%β | Local bf16 diagnostics; public-231 6.06 points and typed-decisions 1.20 points above Qev 1.0 |
| Qev 1.0 4B, experimental 4,096-token inference | Qwen3.5-4B | β | 75.32% | β | Same weights; long-context diagnostic, not released default |
| Jeff 1.1 Qwen 4B | Qwen3.5-4B | 79.35 local | β | 83.87% / 88.40% / 73.81% | 13.25 Capability points above Qev's local recheck |
| Wald 4B v2.0 | Qwen3.5-4B | 77.1 | β | β | 11.12 points above Qev; some questions use reasoning |
| Torchcast decision 4B v1.1 | Qwen3.5-4B | 73.5 | β | β | 7.52 points above Qev |
| decider 4B v2 | Qwen3.5-4B | 70.8 | β | β | 4.82 points above Qev |
| Open Decider 4B typed specialist | Qwen3.5-4B | β | β | 79.60% | Author-reported; trained on benchmark workflows |
| Jeff 1.0 Qwen 4B | Qwen3.5-4B | 68.0 | β | β | 2.02 points above Qev; model-card figure |
| Qev 1.0 4B | Qwen3.5-4B | 65.98 | 68.40% | 64.50%β | Our development result |
| nanohunch decider 4B | Qwen3-4B | 57.1 | β | β | 8.88 points below Qev; four-seed mean, different Qwen generation |
The 82.2 and 79.3 results are reported in Jeff 1.1's same-set evaluation table. The OpenDecisions leaderboard reports 67.7 for Jeff 1.0, versus 68.0 on its model card. Different service configurations can affect results; differences around two points should not be treated as decisive. Our 4B target against the identified 82.2 reference is at least 78.09 Capability (95% of 82.2); Qev 1.0 does not meet it. This is a provisional validation target, not proof of generalization. Kev's local 0.8B result gives a provisional 95%-of-reference target of 56.19 Capability; the experimental pilot at 50.45 falls 5.73 points short using unrounded scores. The local Valen 2B result gives a provisional 95%-of-reference target of 59.02 Capability; Qev 1.0 at 55.23 is 3.79 points short. None of these references is established as global same-size SOTA.
The 2B OpenDecisions comparison uses 2,352 identical inputs and the published scorer: Valen's local run scored 62.12 Capability (48.67 Intelligence, 75.57 Calibration) versus Qev's 55.23 Capability, an 11.10% relative gap. Valen's current local weights match its pinned Hub revision after the run; hashes were not captured at both run boundaries. The native request renderings differ, Valen's model card says its training data is not public, and source overlap and scorer parity remain unverified. This is a public development comparison, not independent generalization evidence.
On the separate Atlan Decision Bench 949 text-only cases, Qev 1.0 4B's local diagnostic scored 749/949 (78.93%), versus Kev 4B's published run 795/949 (83.77%): a 4.85 percentage-point, 5.79% relative deficit. Qev answered 912 and refused 37 overlength cases; Kev answered all 949. On those 912 shared answered cases, Qev/Kev hard-gold Brier was 0.269/0.229 and NLL 0.488/0.431. Qev used its unchanged checkpoint, 2,048-token default and native local candidate rendering; Kev used a hosted OpenRouter checkpoint. Both received the benchmark's judgment-policy wording, but prompts, runtimes and potentially precision differ. State leakage, source overlap and full scorer parity remain unverified; this public Choice-only diagnostic is not an independent generalization result or an official Qev leaderboard run.
Download counts are Hugging Face's rolling monthly counts observed 12 November 2026 for the linked original repositories, not deduplicated people, quality measurements or sums across quantizations. Kev and Decider are included because of their adoption and relevance; popularity itself does not establish they are the strongest on this benchmark.
We are pursuing fresh, leakage-screened paired evaluations and improved training at all three sizes. Existing Qev training source provenance, rights and independent generalization remain unresolved; see the individual model cards. We will update these tables when comparable, reproducible evidence is available.
The evaluation plan targets at least three typed-decision benchmarks: OpenDecisions, JevBench's public 231-item subset, and typed-decisions (2,000 test questions). The public-231 diagnostic results above are accuracy, not OpenDecisions Capability; all three Qev typed-decisions results are preliminary local diagnostics. Typed-decisions uses teacher-derived targets; some competing models train on its training split, so distinguish specialists from generalists. None of the OpenDecisions gaps can be transferred to another benchmark. The public 231 subset is not the full sealed JevBench leaderboard.
For the same 2,000 typed-decisions questions, Kev 0.8B scored 883/2,000 (44.15%) and Qev 1.0 0.8B scored 870/2,000 (43.50%); all were answered. Kev/Qev Brier against teacher distributions was 0.205/0.257 and KL 0.362/0.455 (lower is better). A 2,000-draw paired bootstrap by 400 source scenarios gives Qev minus Kev accuracy an exploratory 95% interval of β3.45 to +2.20 percentage points. Kev used its native float32 predictor and Qev its native scorer, with different prompt renderings; the Windows Kev runtime uses a shim. Teacher quality, scorer parity, training overlap, and independent generalization remain unverified. This comparison does not establish a statistically decisive accuracy winner.
On those same typed-decisions cases, Kev 4B scored 1,314/2,000 (65.70%) and Qev 1.0 4B scored 1,290/2,000 (64.50%); both answered all cases. Kev/Qev Brier against teacher distributions was 0.124/0.137 and KL 0.219/0.245. By type, Qev/Kev were 387/387 correct on 600 Choice, 407/432 on 600 Noul and 496/495 on 800 Score. The paired 2,000-draw, 400-scenario bootstrap interval for Qev minus Kev accuracy is β3.05 to +0.70 percentage points. Kev used its native bf16 predictor on Windows, versus Qev's native local scorer; prompts and precision differ, and teacher labels, scorer parity and source overlap remain unverified. These public-development results do not establish independent generalization or a definitive winner.
Valen 1.5 2B scored 1,383/2,000 (69.15%) versus Qev 1.0 2B's 1,122/2,000 (56.10%), both answering all cases. Valen/Qev Brier against teacher distributions was 0.112/0.208 and KL 0.222/0.388. By type, Qev/Valen were 339/426 correct on 600 Choice, 381/456 on 600 Noul and 402/501 on 800 Score. The paired 2,000-draw, 400-scenario bootstrap interval for Qev minus Valen accuracy is β15.75 to β10.55 percentage points. The native model implementations and prompts differ, local Valen weights were not independently hashed, and teacher-label quality, scorer parity and source overlap remain unverified. This public development comparison is not independent generalization evidence.
Public-231 figures are preliminary local diagnostics on the same 231 published items, matched by ID to the pinned upstream source; accuracy is correct / 231, with unsupported Qev cases counted wrong. Qev 0.8B/2B/4B each refused 36 overlength inputs at their 2,048-token limit and answered 195; their answered-only accuracies were 66.67% / 75.38% / 81.03%. Valen 2B answered all 231. At 95% of Valen's 81.39%, the provisional 2B target on this suite is 77.32%; Qev 1.0 is 13.68 points below that target. We used each model's native decision interface and different prompt renderings; the Qev and Valen score adapters have not been independently certified against the upstream benchmark scorer, and training/source overlap has not been fully audited. The 17.75-point gap is a provisional local observation, not a leaderboard rank or uncontaminated estimate.
Kev 0.8B answered 231/231 and scored 146/231 (63.20%) in a local run of its native predictor at Hub revision bf75a6a8848ea6960ff2ed108d9ed44c2941174f and upstream code revision 2074cc44256200bb37540103fa4572901faa10f3. It is 16 cases, or 6.93 percentage points, above Qev 1.0 0.8B's 130/231 (56.28%); values rounded separately can differ by 0.01 point. Kev used an 8,192-token state limit and Qev 2,048, so Qev's 36 refusals affect the gap. Kev's predictor ran on Windows with a compatibility shim and package versions outside its declared runtime; prompt rendering, scorer parity and training overlap need further verification. The unreleased pilot at an experimental 4,096-token limit scored 142/231 (61.47%), four cases behind Kev, within 5% of this single observed reference score only; it does not establish the multi-benchmark SOTA goal. The pilot was trained at a shorter context and one previously answered prediction flipped between runs.
Mapika Decider 2B answered 231/231 and scored 179/231 (77.49%) in a local text-only run at Hub revision 7614a6c7ea5a047aa7d7bcfb1a0015469381c3e3, upstream code revision 91b6fa9b94ab6c4dc2c63d3b14d413243b4fffcc, bf16 and an 8,192-token state limit. It is 32 cases (13.85 percentage points) above Qev 1.0 2B's 147/231. We used the model card's separately fitted Choice/Noul/Score temperatures 2.465/3.185/2.833. Its shipped Noul default is 0.05, which the author says improves a benchmark-specific rule at the cost of calibration; this result uses the fitted setting. Its native API rounds probabilities to four decimals before our normalization. The author reports using benchmark readouts in checkpoint selection; model rendering, context, overlap and scorer parity differ or remain unverified. These are preliminary local diagnostics, not an independent ranking.
Broader 2B diagnostic on public-231: Qev 1.0 answered 195/231 (84.42% coverage), while Valen and Decider each answered all 231. On the same 195 cases that all answered, hard-gold multiclass Brier was 0.330 for Qev, 0.205 for Valen and 0.247 for Decider (lower is better); NLL was 0.572, 0.349 and 0.442 respectively. These probability metrics exclude Qev's 36 refusals and do not establish that the gold labels represent true uncertainty. An exploratory 2,000-replicate cluster bootstrap of all-item accuracy gives a 95% interval for Qev minus Valen of β24.0 to β11.6 percentage points, and Qev minus Decider of β19.9 to β7.5; it reflects this public development sample, not an independent final test. Different native interfaces and context limits remain material. Comparable latency, memory, independent OOD robustness and semantic-overlap audits are not yet available; no all-around winner is established by these figures.
Kev 4B scored 172/231 (74.46%) versus Qev 1.0 4B's 158/231 (68.40%); Kev answered all items and Qev refused 36 overlength cases. On the 195 cases both answered, hard-gold Brier was 0.261 for Qev and 0.262 for Kev, and NLL was 0.468 and 0.457 respectively (lower is better). The exploratory 2,000-replicate cluster-bootstrap 95% interval for Qev minus Kev all-item accuracy is β12.2 to 0.0 percentage points. Kev's pinned revision 6cfce5c2fa4b4bd64026336ab649c5ca78857d52 ran through its native predictor in bf16 on Windows, with an 8,192-token state limit; its published evaluation uses fp32. Qev uses a 2,048-token limit and a different rendering. Scorer parity, source overlap and independent generalization are unverified. On a separate small synthetic maximum-selection diagnostic, Kev scored 96/96 and Qev 70/96; neither changed answers on the 24 option-reorder or 24 irrelevant-text pairs. This single artificial task is not a general ranking and related task families may occur in training.
An inference-only Qev 4B experiment raised the limit to 4,096 tokens without truncation or changing its weights or temperature. It answered all 231 cases and scored 174/231 (75.32%); Kev scored 172/231. On all 231, Qev/Kev hard-gold Brier was 0.307/0.323, NLL 0.557/0.569, and 10-bin top-label ECE 0.074/0.041, respectively (lower is better). The exploratory paired cluster-bootstrap 95% interval for Qev minus Kev accuracy was β4.5 to +6.2 percentage points. One previously answered Qev prediction changed between runs. The longer inference setting exceeds Qev's training limit, is not its released default, and on the long-input run total GPU usage reached about 15.7 GiB on the local 16 GiB card. The public cases are development diagnostics with different native renderings, unknown overlap and unverified scorer parity; the small accuracy difference does not establish an overall winner or long-context generalization.
Exploratory batch-1 timings on the same 24 short synthetic inputs on an RTX 5070 Ti after three warmups: Qev 4B median/p95 93.71/100.46 ms and peak allocated tensor memory 7.97 GiB; Kev 4B 82.51/98.79 ms and 8.04 GiB. Timings used native implementations with different prompt rendering and were measured in separate runs; Qev's run coincided with a background download. Both are small, unoptimized local measurements, not production latency or definitive speed rankings.
On OpenDecisions, the same pinned Kev native predictor answered all 2,352 items and scored 59.14 Capability (31.18 intelligence, 87.10 calibration), compared with Qev 1.0 0.8B at 46.85 and the unreleased pilot at 50.45. These are development diagnostics, not independently held-out results; the same runtime, rendering, scorer-parity and overlap limitations apply. Capability is not accuracy. The official local scorer was applied to the recorded keyed probability vectors.
An earlier unreleased 0.8B checkpoint trained on a broad Tasksource-derived mixture scored 49.32 OpenDecisions Capability and 133/231 (57.58%) on public-231; it refused the same 36 long inputs as Qev 1.0. On the 195 cases both answered, its hard-gold Brier/NLL were 0.458/0.821 versus Qev 1.0's 0.418/0.703 (lower is better). On a small generated invariance diagnostic it scored 41/96 versus Qev 1.0's 63/96; its prior training included related procedural tasks. These are diagnostic tradeoffs, not an independent generalization result. Source rights and semantic overlap remain unresolved, and this checkpoint is not promoted.
β Typed-decisions: Qev 0.8B scored 870/2,000 decisions (43.50%), 2B 1,122/2,000 (56.10%), and 4B 1,290/2,000 (64.50%), using argmax of returned distributions; none refused a case. Qev used each question separately, and 200 Noul inputs without criteria were supplied explicit neutral No/Yes definitions. This conversion and our accuracy policy have not been independently certified against the official scorer. Brier/KL against teacher-derived soft targets were 0.2567/0.4553 for 0.8B, 0.2079/0.3884 for 2B, and 0.1369/0.2447 for 4B; our calculation reproduces the benchmark uniform Brier reference at 0.2382. The publisher reports a 47.0% prior baseline and an independently reported 0.8B workflow specialist at 75.1%, but the latter and the 4B specialist trained on the benchmark training workflows and used different request formatting. The published numbers are context, not controlled head-to-head results. Dataset; specialist model card.
The local 0.8B pilot is a separate one-pass LoRA and head trained on 1,000 teacher-target decisions from the typed-decisions training workflows, with temperature fitted on separate scenarios. Its held-out typed test score is 1,163/2,000, with Brier 0.1377 and KL 0.2366. It scored 50.45 Capability on all 2,352 OpenDecisions development items and 130/231 on public-231; the same 36 overlength inputs were counted incorrect. Exact normalized scenario matches across its training and these evaluations were absent, but semantic overlap, scorer equivalence, rights and independent generalization remain unverified. It is not a released model or a clean independent benchmark result; the specialist's reported 75.1% uses a different setup.
The separate local 2B one-pass pilot used the same 1,000 teacher-target workflow decisions and separate calibration scenarios. It scored 51.93 OpenDecisions Capability versus Qev 1.0 2B's 55.23, and 1,388/2,000 (69.40%) on held-out typed-workflow scenarios versus 56.10% for Qev 1.0. Typed Brier/KL against teacher targets were 0.0925/0.1739. On public-231 it answered all cases at an experimental 4,096-token inference limit and scored 162/231 (70.13%), versus Qev 1.0's 147/231 (63.64%) and Valen's 188/231 (81.39%). This is a mixed outcome with an OpenDecisions regression: the workflow test is related to training, the longer context exceeds the training limit, rendering differs, overlap and scorer parity are unverified, and no independent generalization or release claim follows. Future training prioritizes broad task generalization rather than tuning to these evaluation cases.
Typed-decisions peer comparison (exploratory)
On the public LocalLLaMA typed-decisions 2,000-item test, Qev scored 64.50%; the same-size reference scored 65.70%. Qev's paired accuracy difference is β1.83% relative (β1.20 percentage points; paired 95% scenario-cluster interval β3.20 to +0.75 points).
This is an exploratory comparison on teacher-derived labels, not an independent SOTA ranking. The models used different native interfaces and prompt renderings; scorer equivalence and training-source overlap are unresolved. The scenario-cluster interval reflects sampling variability only and does not account for these systematic differences. Qev has not established that it is within 5% of the strongest model across benchmarks. See the full three-size, multi-benchmark comparison.
Atlan Decision Bench comparison (exploratory)
On the public Atlan Decision Bench text-only set, Qev 1.0 4B scored 749/949 (78.93%); Kev 4B scored 795/949 (83.77%) in the benchmark's published run. This is a 5.79% relative deficit to Kev, beyond the goal's 5% relative gap; the 95% threshold is 79.58% (at least 756 correct of 949). Qev answered 912 cases and refused 37 overlength inputs, while Kev answered all 949. On Qev's 912 answered cases, Qev/Kev hard-label Brier was 0.269/0.229 and NLL 0.488/0.431 (lower is better).
This is a public Choice-only diagnostic, not an independent test or official Qev leaderboard run. Qev used its unchanged checkpoint, 2,048-token default and native local rendering; Kev used a hosted OpenRouter checkpoint. Both received the benchmark policy wording, but prompts, runtimes and precision may differ. State leakage, training-source overlap and full scorer parity remain unverified. See the paired comparison and Kev run and the Atlan Decision Bench.
Preliminary cross-model diagnostic: This-That Complex Decisions (2026-11-13)
This is a single public Choice-suite diagnostic, not an overall ranking or independent generalization result. It uses all 1,710 items across 19 generated policy families from limberc/this-that-complex-decisions revision b77207bd24ad096ee83b1e1b026d2f130259d616. No benchmark questions were used to train, calibrate, select checkpoints, or tune prompts. Accuracy is higher-is-better; NLL, Brier, and ECE are lower-is-better. ECE is occupancy-weighted, equal-width 15-bin top-label ECE.
0.8B
| Model | Parameters | Accuracy | NLL | Multiclass Brier | 15-bin ECE | HF downloads, prior month |
|---|---|---|---|---|---|---|
| Qev 1.0 0.8B | 0.8B | 32.40% | 1.516 | 0.753 | 0.10406 | 0 |
| Kev 0.8B | 0.8B | Not evaluated on this suite | β | β | β | 14,032 |
2B
| Model | Parameters | Accuracy | NLL | Multiclass Brier | 15-bin ECE | HF downloads, prior month |
|---|---|---|---|---|---|---|
| Qev 1.0 2B | 2B | 31.70% | 1.657 | 0.784 | 0.15145 | 0 |
| Decider 2B | 2B | 39.59% local | 1.462 | 0.736 | 0.13571 | 7,391 |
| FLock this-that-model-1.1 | 1.88B | 77.84% local (77.5% card) | 0.609 | 0.303 | 0.06891 | 34 |
4B
| Model | Parameters | Accuracy | NLL | Multiclass Brier | 15-bin ECE | HF downloads, prior month |
|---|---|---|---|---|---|---|
| Qev 1.0 4B | 4B | 42.63% | 1.371 | 0.706 | 0.11397 | 0 |
| Kev 4B | 4B | Not evaluated on this suite | β | β | β | 12,499 |
Decider 2B was run through its native Decider.decide API at bf16 and its stored temperature (2.515). Against it, Qev reaches 80.06% of Decider's accuracy, below the 95% target and 5.92 pp below the 37.61% acceptance threshold. Qev has worse NLL, Brier and corrected ECE. A 50,000-replicate family-cluster bootstrap interval for the accuracy gain is +3.10 to +12.81 pp for Decider; it beat Qev on 14 of 19 families, tied one, and lost four. Qev scores each option separately using its fitted temperature, so serving/scoring differences limit direct calibration and latency comparison. FLock 1.1 used its official single-pass TypedDecider API at bf16 and temperature 1.0; its overlap with this benchmark's rule structures is unverified. Newer FLock 1.2 is excluded because its card lists this exact benchmark as training data.
Metric correction (2026-11-13): An earlier report understated the two local peer ECE values because it omitted bin-occupancy weighting. ECE was recalculated from retained per-item probability vectors using the same occupancy-weighted formula as Qev's evaluator. FLock 1.1 is 0.06891 (supersedes 0.00075), and Decider 2B is 0.13571 (supersedes 0.00109). Qev's values were independently recomputed and are unchanged. Accuracy, NLL, and Brier did not change.
Kev 0.8B and 4B same-suite runs were stopped before scoring any benchmark rows because the pinned official predictor imports POSIX fcntl, unavailable in this Windows runtime; no Kev results are claimed here. Popularity counts are Hugging Face monthly downloads observed 2026-11-13, not unique users or evidence of quality. All comparisons remain single-suite diagnostics; see the Qev comparison report for the other pinned suites, strong popular peers, and their gaps. The multi-suite, independent evaluation and rights gates remain open.
- Downloads last month
- -