Qev1.0-Qwen3.5-4B

Qev 1.0 4B is an experimental decision model built on Qwen3.5-4B. It scores candidate answers to Choice, Noul and Score questions. This repository contains a LoRA adapter, a separate learned decision head and calibration metadata; it requires the base model and the Open-Jev decision-model implementation. It is not a standalone chat model.

Intended use and limitations

Use this checkpoint for research and evaluation of structured decisions. Test it on representative tasks before using it in a router or consequential workflow. The development panel is provisional and not frozen. Independent generalization, reliable calibration across domains, cost savings and parity with other decision models have not been established. Outputs can be wrong; do not treat a score as a guarantee.

Evaluation

On the 2,352-item OpenDecisions validation set used during development, this checkpoint obtained a capability score of 65.98, with Choice, Noul and Score accuracies of 68.25%, 77.30% and 56.46%, respectively. These are development-set results, not a fresh independent test. The source benchmark and scoring definitions are described in OpenDecisions. Results may be affected by model-selection and data-overlap limitations.

4B all-around OpenDecisions profile (local, 13 November 2026)

Both pinned checkpoints were run on the same 2,352 OpenDecisions 1.0.1 items (dataset SHA-256 78614972006c5e5efbbfc121ce31e15cf576ef6c0cdf24a229b33d34d2bfcda6). This is public development data, not independent generalization evidence. Jeff's model card states that OpenDecisions informed training-length selection. The model-specific native prompts and inference paths differ, so these results do not isolate the effect of training.

Metric Qev 1.0 4B recheck Jeff 1.1 Qwen 4B Observed leader
Capability / Intelligence / Calibration 66.09 / 44.70 / 87.49 79.35 / 68.46 / 90.23 Jeff / Jeff / Jeff
Choice / Noul / Score accuracy 68.34% / 77.13% / 56.46% 83.87% / 88.40% / 73.81% Jeff / Jeff / Jeff
Choice / Noul / Score competence 59.23 / 49.39 / 25.48 79.57 / 74.55 / 51.25 Jeff / Jeff / Jeff
Choice / Noul / Score calibration 80.70 / 91.71 / 90.05 88.67 / 92.40 / 89.63 Jeff / Jeff / Qev
Calibration-target TVD, 129 items (lower better) 0.18760 0.08092 Jeff
Hard-label NLL / multiclass Brier (lower better) 0.79124 / 0.43067 0.48293 / 0.25889 Jeff / Jeff
Top-label ECE, 10 bins (lower better) 0.02177 0.03816 Qev, point estimate
Coverage / refusals 100% / 0 100% / 0 Tie

Paired family-cluster bootstrap (5,000 draws, 11 families): Qev minus Jeff Capability βˆ’13.25 points (95% CI βˆ’18.96 to βˆ’10.44), Intelligence βˆ’23.76 (βˆ’31.83 to βˆ’19.46), and Calibration βˆ’2.75 (βˆ’9.47 to +1.22). Accuracy differences were Choice βˆ’15.53 pp (βˆ’23.67 to βˆ’11.57), Noul βˆ’11.26 pp (βˆ’18.13 to βˆ’6.58), and Score βˆ’17.35 pp (βˆ’26.06 to βˆ’12.47). Supplementary paired differences were NLL +0.308 (+0.255 to +0.403), Brier +0.172 (+0.145 to +0.222), ECE βˆ’0.016 (βˆ’0.030 to +0.020; includes zero), and TVD +0.107 (+0.106 to +0.107). The TVD interval is conditional on bootstrap samples containing calibration-target items.

Latency and GPU memory were measured on the same RTX 5070 Ti but through different native serving protocols; they are descriptive, not controlled efficiency comparisons. The full method, model-specific hashes, row-level outputs and all intervals are linked in the Qev comparison report. Qev leads two calibration point estimates, while Jeff leads most measured quality metrics. Neither this profile nor a single Capability score establishes overall SOTA or generalization.

Training and provenance

The hybrid-v2 checkpoint used 1,000 selected decisions (400 Choice, 300 Noul, 300 Score), 250 optimizer steps, rank-8 LoRA and a separate 128-row calibration split. Training input SHA-256: a7f4c474442e8f577765630649e893324cd09257420c3a3b11f6c98d28939c25. The base revision is 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a.

The selected mixture includes 100 CommonsenseQA Choice examples, 100 Banking77 Choice controls, 150 Measuring Hate Speech Noul examples and 150 HelpSteer2 Score examples; the remainder includes Open-Jev-derived controls. The selection is not independently adjudicated or frozen. The HelpSteer2 dataset card lists CC-BY-4.0. The selected prompts' individual upstream origins have not been resolved. The upstream Qwen base card lists Apache-2.0. These facts do not independently establish the license of the adapter or all training-data permissions.

Files and loading

  • adapter/adapter_model.safetensors and adapter/adapter_config.json: LoRA adapter.
  • head.pt: separate learned decision head.
  • temperature.json: calibration temperature; applied separately by the evaluation code.
  • model.json: model and training configuration.
  • SHA256SUMS.json: hashes of the staged model files.

The checked loader comes from Open-Jev commit bd4118882f733574a3250a4b65fe4d884130c08b. Its source is MIT licensed. Install that revision and its training dependencies, then point the loader at the downloaded checkpoint directory:

git clone https://github.com/Zefan-Cai/Open-Jev.git
cd Open-Jev
git checkout bd4118882f733574a3250a4b65fe4d884130c08b
pip install -e '.[train]'
from pathlib import Path
from jev.model import DecisionModel

checkpoint = Path("/path/to/downloaded/qev-checkpoint")
model = DecisionModel.load(checkpoint, device="cpu")

DecisionModel.load loads the base model, adapter and head; it does not apply temperature.json automatically. A local CPU load check succeeded with an existing local base copy and the staged adapter and head. A fresh Hub download, base-file hashes, clean-environment setup and inference quality were not verified by that check.

A local CPU smoke used a synthetic yes/no question and returned finite uncalibrated logits approximately [0.0, 1.8188] with the local base copy. This single observation is not an accuracy, calibration or clean-download test.

Qev 1.0 comparison by model size

Qev 1.0: comparison with same-size decision models

Snapshot: 13 November 2026. Qev's OpenDecisions results use a 2,352-item development set, not a fresh independent test. Capability averages the benchmark's intelligence and calibration scores; it is not accuracy. Public-231 accuracy is a separate local diagnostic detailed below. Unless marked local, competitor scores are reported by their authors or the benchmark authors using their own serving configurations. Tables list identified comparators, not a complete ranking or proof of global state of the art. Different benchmarks cannot be subtracted.

0.8B

Model Base OpenDecisions Capability Public 231 accuracy Typed-decisions accuracy Comparison to Qev 1.0 0.8B
Qev 1.0 0.8B Qwen3.5-0.8B 46.85 56.28% 43.50%† Our development result
Experimental local 0.8B pilot Qwen3.5-0.8B 50.45 56.28% 58.15%† Unreleased workflow-trained pilot; preliminary, not independently accepted
Experimental Tasksource 0.8B Qwen3.5-0.8B 49.32 57.58% β€” Unreleased broad-source checkpoint; probability quality and robustness tradeoffs
Kev 0.8B Qwen3.5-0.8B-Base 59.14 63.20% 44.15%† Local diagnostics: 12.29 Capability points and 6.93 public-231 accuracy points above Qev 1.0
StartLux 1.1 0.8B Instant Qwen3.5-0.8B Not verified β€” β€” Gap unknown; same-suite run needed
AA Decision 0.8B Qwen3.5-0.8B Not verified β€” β€” Gap unknown; card reports different benchmarks
labqoat QEV 0.8B Qwen3.5-0.8B Not verified β€” β€” Different project; gap unknown

2B

Model Base OpenDecisions Capability Public 231 accuracy Typed-decisions accuracy Comparison to Qev 1.0 2B
Qev 1.0 2B Qwen3.5-2B 55.23 63.64% 56.10%† Our development result
Experimental local 2B pilot Qwen3.5-2B 51.93 70.13% 69.40%† Unreleased workflow-trained pilot; OpenDecisions regressed
Valen Decision 1.5 2B Qwen3.5-2B 62.12 81.39% 69.15%† Local runs: 6.90 Capability points, 17.75 public-231 and 13.05 typed-decisions accuracy points above Qev 2B
Decider 2B Qwen3.5-2B-Base Not verified 77.49% β€” Local calibrated public-231 run; 13.85 points above Qev 1.0
StartLux 1.1 2B Instant Qwen3.5-2B Not verified β€” β€” Gap unknown; same-suite run needed
joey-1 Qwen3.5-2B Not verified β€” β€” Gap unknown; same-suite run needed
Jev-Style 2B Decision v3 Qwen3.5-2B Not verified β€” β€” Gap unknown; same-suite run needed

4B

Model Base OpenDecisions Capability Public 231 accuracy Typed-decisions accuracy Comparison to Qev 1.0 4B
StartLux 1.1 4B Instant Qwen3.5-4B 82.2 β€” β€” 16.22 points above Qev; 19.73% relative gap
Kev 4B Qwen3.5-4B-Base Not verified 74.46% 65.70%† Local bf16 diagnostics; public-231 6.06 points and typed-decisions 1.20 points above Qev 1.0
Qev 1.0 4B, experimental 4,096-token inference Qwen3.5-4B β€” 75.32% β€” Same weights; long-context diagnostic, not released default
Jeff 1.1 Qwen 4B Qwen3.5-4B 79.35 local β€” 83.87% / 88.40% / 73.81% 13.25 Capability points above Qev's local recheck
Wald 4B v2.0 Qwen3.5-4B 77.1 β€” β€” 11.12 points above Qev; some questions use reasoning
Torchcast decision 4B v1.1 Qwen3.5-4B 73.5 β€” β€” 7.52 points above Qev
decider 4B v2 Qwen3.5-4B 70.8 β€” β€” 4.82 points above Qev
Open Decider 4B typed specialist Qwen3.5-4B β€” β€” 79.60% Author-reported; trained on benchmark workflows
Jeff 1.0 Qwen 4B Qwen3.5-4B 68.0 β€” β€” 2.02 points above Qev; model-card figure
Qev 1.0 4B Qwen3.5-4B 65.98 68.40% 64.50%† Our development result
nanohunch decider 4B Qwen3-4B 57.1 β€” β€” 8.88 points below Qev; four-seed mean, different Qwen generation

The 82.2 and 79.3 results are reported in Jeff 1.1's same-set evaluation table. The OpenDecisions leaderboard reports 67.7 for Jeff 1.0, versus 68.0 on its model card. Different service configurations can affect results; differences around two points should not be treated as decisive. Our 4B target against the identified 82.2 reference is at least 78.09 Capability (95% of 82.2); Qev 1.0 does not meet it. This is a provisional validation target, not proof of generalization. Kev's local 0.8B result gives a provisional 95%-of-reference target of 56.19 Capability; the experimental pilot at 50.45 falls 5.73 points short using unrounded scores. The local Valen 2B result gives a provisional 95%-of-reference target of 59.02 Capability; Qev 1.0 at 55.23 is 3.79 points short. None of these references is established as global same-size SOTA.

The 2B OpenDecisions comparison uses 2,352 identical inputs and the published scorer: Valen's local run scored 62.12 Capability (48.67 Intelligence, 75.57 Calibration) versus Qev's 55.23 Capability, an 11.10% relative gap. Valen's current local weights match its pinned Hub revision after the run; hashes were not captured at both run boundaries. The native request renderings differ, Valen's model card says its training data is not public, and source overlap and scorer parity remain unverified. This is a public development comparison, not independent generalization evidence.

On the separate Atlan Decision Bench 949 text-only cases, Qev 1.0 4B's local diagnostic scored 749/949 (78.93%), versus Kev 4B's published run 795/949 (83.77%): a 4.85 percentage-point, 5.79% relative deficit. Qev answered 912 and refused 37 overlength cases; Kev answered all 949. On those 912 shared answered cases, Qev/Kev hard-gold Brier was 0.269/0.229 and NLL 0.488/0.431. Qev used its unchanged checkpoint, 2,048-token default and native local candidate rendering; Kev used a hosted OpenRouter checkpoint. Both received the benchmark's judgment-policy wording, but prompts, runtimes and potentially precision differ. State leakage, source overlap and full scorer parity remain unverified; this public Choice-only diagnostic is not an independent generalization result or an official Qev leaderboard run.

Download counts are Hugging Face's rolling monthly counts observed 12 November 2026 for the linked original repositories, not deduplicated people, quality measurements or sums across quantizations. Kev and Decider are included because of their adoption and relevance; popularity itself does not establish they are the strongest on this benchmark.

We are pursuing fresh, leakage-screened paired evaluations and improved training at all three sizes. Existing Qev training source provenance, rights and independent generalization remain unresolved; see the individual model cards. We will update these tables when comparable, reproducible evidence is available.

The evaluation plan targets at least three typed-decision benchmarks: OpenDecisions, JevBench's public 231-item subset, and typed-decisions (2,000 test questions). The public-231 diagnostic results above are accuracy, not OpenDecisions Capability; all three Qev typed-decisions results are preliminary local diagnostics. Typed-decisions uses teacher-derived targets; some competing models train on its training split, so distinguish specialists from generalists. None of the OpenDecisions gaps can be transferred to another benchmark. The public 231 subset is not the full sealed JevBench leaderboard.

For the same 2,000 typed-decisions questions, Kev 0.8B scored 883/2,000 (44.15%) and Qev 1.0 0.8B scored 870/2,000 (43.50%); all were answered. Kev/Qev Brier against teacher distributions was 0.205/0.257 and KL 0.362/0.455 (lower is better). A 2,000-draw paired bootstrap by 400 source scenarios gives Qev minus Kev accuracy an exploratory 95% interval of βˆ’3.45 to +2.20 percentage points. Kev used its native float32 predictor and Qev its native scorer, with different prompt renderings; the Windows Kev runtime uses a shim. Teacher quality, scorer parity, training overlap, and independent generalization remain unverified. This comparison does not establish a statistically decisive accuracy winner.

On those same typed-decisions cases, Kev 4B scored 1,314/2,000 (65.70%) and Qev 1.0 4B scored 1,290/2,000 (64.50%); both answered all cases. Kev/Qev Brier against teacher distributions was 0.124/0.137 and KL 0.219/0.245. By type, Qev/Kev were 387/387 correct on 600 Choice, 407/432 on 600 Noul and 496/495 on 800 Score. The paired 2,000-draw, 400-scenario bootstrap interval for Qev minus Kev accuracy is βˆ’3.05 to +0.70 percentage points. Kev used its native bf16 predictor on Windows, versus Qev's native local scorer; prompts and precision differ, and teacher labels, scorer parity and source overlap remain unverified. These public-development results do not establish independent generalization or a definitive winner.

Valen 1.5 2B scored 1,383/2,000 (69.15%) versus Qev 1.0 2B's 1,122/2,000 (56.10%), both answering all cases. Valen/Qev Brier against teacher distributions was 0.112/0.208 and KL 0.222/0.388. By type, Qev/Valen were 339/426 correct on 600 Choice, 381/456 on 600 Noul and 402/501 on 800 Score. The paired 2,000-draw, 400-scenario bootstrap interval for Qev minus Valen accuracy is βˆ’15.75 to βˆ’10.55 percentage points. The native model implementations and prompts differ, local Valen weights were not independently hashed, and teacher-label quality, scorer parity and source overlap remain unverified. This public development comparison is not independent generalization evidence.

Public-231 figures are preliminary local diagnostics on the same 231 published items, matched by ID to the pinned upstream source; accuracy is correct / 231, with unsupported Qev cases counted wrong. Qev 0.8B/2B/4B each refused 36 overlength inputs at their 2,048-token limit and answered 195; their answered-only accuracies were 66.67% / 75.38% / 81.03%. Valen 2B answered all 231. At 95% of Valen's 81.39%, the provisional 2B target on this suite is 77.32%; Qev 1.0 is 13.68 points below that target. We used each model's native decision interface and different prompt renderings; the Qev and Valen score adapters have not been independently certified against the upstream benchmark scorer, and training/source overlap has not been fully audited. The 17.75-point gap is a provisional local observation, not a leaderboard rank or uncontaminated estimate.

Kev 0.8B answered 231/231 and scored 146/231 (63.20%) in a local run of its native predictor at Hub revision bf75a6a8848ea6960ff2ed108d9ed44c2941174f and upstream code revision 2074cc44256200bb37540103fa4572901faa10f3. It is 16 cases, or 6.93 percentage points, above Qev 1.0 0.8B's 130/231 (56.28%); values rounded separately can differ by 0.01 point. Kev used an 8,192-token state limit and Qev 2,048, so Qev's 36 refusals affect the gap. Kev's predictor ran on Windows with a compatibility shim and package versions outside its declared runtime; prompt rendering, scorer parity and training overlap need further verification. The unreleased pilot at an experimental 4,096-token limit scored 142/231 (61.47%), four cases behind Kev, within 5% of this single observed reference score only; it does not establish the multi-benchmark SOTA goal. The pilot was trained at a shorter context and one previously answered prediction flipped between runs.

Mapika Decider 2B answered 231/231 and scored 179/231 (77.49%) in a local text-only run at Hub revision 7614a6c7ea5a047aa7d7bcfb1a0015469381c3e3, upstream code revision 91b6fa9b94ab6c4dc2c63d3b14d413243b4fffcc, bf16 and an 8,192-token state limit. It is 32 cases (13.85 percentage points) above Qev 1.0 2B's 147/231. We used the model card's separately fitted Choice/Noul/Score temperatures 2.465/3.185/2.833. Its shipped Noul default is 0.05, which the author says improves a benchmark-specific rule at the cost of calibration; this result uses the fitted setting. Its native API rounds probabilities to four decimals before our normalization. The author reports using benchmark readouts in checkpoint selection; model rendering, context, overlap and scorer parity differ or remain unverified. These are preliminary local diagnostics, not an independent ranking.

Broader 2B diagnostic on public-231: Qev 1.0 answered 195/231 (84.42% coverage), while Valen and Decider each answered all 231. On the same 195 cases that all answered, hard-gold multiclass Brier was 0.330 for Qev, 0.205 for Valen and 0.247 for Decider (lower is better); NLL was 0.572, 0.349 and 0.442 respectively. These probability metrics exclude Qev's 36 refusals and do not establish that the gold labels represent true uncertainty. An exploratory 2,000-replicate cluster bootstrap of all-item accuracy gives a 95% interval for Qev minus Valen of βˆ’24.0 to βˆ’11.6 percentage points, and Qev minus Decider of βˆ’19.9 to βˆ’7.5; it reflects this public development sample, not an independent final test. Different native interfaces and context limits remain material. Comparable latency, memory, independent OOD robustness and semantic-overlap audits are not yet available; no all-around winner is established by these figures.

Kev 4B scored 172/231 (74.46%) versus Qev 1.0 4B's 158/231 (68.40%); Kev answered all items and Qev refused 36 overlength cases. On the 195 cases both answered, hard-gold Brier was 0.261 for Qev and 0.262 for Kev, and NLL was 0.468 and 0.457 respectively (lower is better). The exploratory 2,000-replicate cluster-bootstrap 95% interval for Qev minus Kev all-item accuracy is βˆ’12.2 to 0.0 percentage points. Kev's pinned revision 6cfce5c2fa4b4bd64026336ab649c5ca78857d52 ran through its native predictor in bf16 on Windows, with an 8,192-token state limit; its published evaluation uses fp32. Qev uses a 2,048-token limit and a different rendering. Scorer parity, source overlap and independent generalization are unverified. On a separate small synthetic maximum-selection diagnostic, Kev scored 96/96 and Qev 70/96; neither changed answers on the 24 option-reorder or 24 irrelevant-text pairs. This single artificial task is not a general ranking and related task families may occur in training.

An inference-only Qev 4B experiment raised the limit to 4,096 tokens without truncation or changing its weights or temperature. It answered all 231 cases and scored 174/231 (75.32%); Kev scored 172/231. On all 231, Qev/Kev hard-gold Brier was 0.307/0.323, NLL 0.557/0.569, and 10-bin top-label ECE 0.074/0.041, respectively (lower is better). The exploratory paired cluster-bootstrap 95% interval for Qev minus Kev accuracy was βˆ’4.5 to +6.2 percentage points. One previously answered Qev prediction changed between runs. The longer inference setting exceeds Qev's training limit, is not its released default, and on the long-input run total GPU usage reached about 15.7 GiB on the local 16 GiB card. The public cases are development diagnostics with different native renderings, unknown overlap and unverified scorer parity; the small accuracy difference does not establish an overall winner or long-context generalization.

Exploratory batch-1 timings on the same 24 short synthetic inputs on an RTX 5070 Ti after three warmups: Qev 4B median/p95 93.71/100.46 ms and peak allocated tensor memory 7.97 GiB; Kev 4B 82.51/98.79 ms and 8.04 GiB. Timings used native implementations with different prompt rendering and were measured in separate runs; Qev's run coincided with a background download. Both are small, unoptimized local measurements, not production latency or definitive speed rankings.

On OpenDecisions, the same pinned Kev native predictor answered all 2,352 items and scored 59.14 Capability (31.18 intelligence, 87.10 calibration), compared with Qev 1.0 0.8B at 46.85 and the unreleased pilot at 50.45. These are development diagnostics, not independently held-out results; the same runtime, rendering, scorer-parity and overlap limitations apply. Capability is not accuracy. The official local scorer was applied to the recorded keyed probability vectors.

An earlier unreleased 0.8B checkpoint trained on a broad Tasksource-derived mixture scored 49.32 OpenDecisions Capability and 133/231 (57.58%) on public-231; it refused the same 36 long inputs as Qev 1.0. On the 195 cases both answered, its hard-gold Brier/NLL were 0.458/0.821 versus Qev 1.0's 0.418/0.703 (lower is better). On a small generated invariance diagnostic it scored 41/96 versus Qev 1.0's 63/96; its prior training included related procedural tasks. These are diagnostic tradeoffs, not an independent generalization result. Source rights and semantic overlap remain unresolved, and this checkpoint is not promoted.

† Typed-decisions: Qev 0.8B scored 870/2,000 decisions (43.50%), 2B 1,122/2,000 (56.10%), and 4B 1,290/2,000 (64.50%), using argmax of returned distributions; none refused a case. Qev used each question separately, and 200 Noul inputs without criteria were supplied explicit neutral No/Yes definitions. This conversion and our accuracy policy have not been independently certified against the official scorer. Brier/KL against teacher-derived soft targets were 0.2567/0.4553 for 0.8B, 0.2079/0.3884 for 2B, and 0.1369/0.2447 for 4B; our calculation reproduces the benchmark uniform Brier reference at 0.2382. The publisher reports a 47.0% prior baseline and an independently reported 0.8B workflow specialist at 75.1%, but the latter and the 4B specialist trained on the benchmark training workflows and used different request formatting. The published numbers are context, not controlled head-to-head results. Dataset; specialist model card.

The local 0.8B pilot is a separate one-pass LoRA and head trained on 1,000 teacher-target decisions from the typed-decisions training workflows, with temperature fitted on separate scenarios. Its held-out typed test score is 1,163/2,000, with Brier 0.1377 and KL 0.2366. It scored 50.45 Capability on all 2,352 OpenDecisions development items and 130/231 on public-231; the same 36 overlength inputs were counted incorrect. Exact normalized scenario matches across its training and these evaluations were absent, but semantic overlap, scorer equivalence, rights and independent generalization remain unverified. It is not a released model or a clean independent benchmark result; the specialist's reported 75.1% uses a different setup.

The separate local 2B one-pass pilot used the same 1,000 teacher-target workflow decisions and separate calibration scenarios. It scored 51.93 OpenDecisions Capability versus Qev 1.0 2B's 55.23, and 1,388/2,000 (69.40%) on held-out typed-workflow scenarios versus 56.10% for Qev 1.0. Typed Brier/KL against teacher targets were 0.0925/0.1739. On public-231 it answered all cases at an experimental 4,096-token inference limit and scored 162/231 (70.13%), versus Qev 1.0's 147/231 (63.64%) and Valen's 188/231 (81.39%). This is a mixed outcome with an OpenDecisions regression: the workflow test is related to training, the longer context exceeds the training limit, rendering differs, overlap and scorer parity are unverified, and no independent generalization or release claim follows. Future training prioritizes broad task generalization rather than tuning to these evaluation cases.

Typed-decisions peer comparison (exploratory)

On the public LocalLLaMA typed-decisions 2,000-item test, Qev scored 64.50%; the same-size reference scored 65.70%. Qev's paired accuracy difference is βˆ’1.83% relative (βˆ’1.20 percentage points; paired 95% scenario-cluster interval βˆ’3.20 to +0.75 points).

This is an exploratory comparison on teacher-derived labels, not an independent SOTA ranking. The models used different native interfaces and prompt renderings; scorer equivalence and training-source overlap are unresolved. The scenario-cluster interval reflects sampling variability only and does not account for these systematic differences. Qev has not established that it is within 5% of the strongest model across benchmarks. See the full three-size, multi-benchmark comparison.

Atlan Decision Bench comparison (exploratory)

On the public Atlan Decision Bench text-only set, Qev 1.0 4B scored 749/949 (78.93%); Kev 4B scored 795/949 (83.77%) in the benchmark's published run. This is a 5.79% relative deficit to Kev, beyond the goal's 5% relative gap; the 95% threshold is 79.58% (at least 756 correct of 949). Qev answered 912 cases and refused 37 overlength inputs, while Kev answered all 949. On Qev's 912 answered cases, Qev/Kev hard-label Brier was 0.269/0.229 and NLL 0.488/0.431 (lower is better).

This is a public Choice-only diagnostic, not an independent test or official Qev leaderboard run. Qev used its unchanged checkpoint, 2,048-token default and native local rendering; Kev used a hosted OpenRouter checkpoint. Both received the benchmark policy wording, but prompts, runtimes and precision may differ. State leakage, training-source overlap and full scorer parity remain unverified. See the paired comparison and Kev run and the Atlan Decision Bench.

Preliminary cross-model diagnostic: This-That Complex Decisions (2026-11-13)

This is a single public Choice-suite diagnostic, not an overall ranking or independent generalization result. It uses all 1,710 items across 19 generated policy families from limberc/this-that-complex-decisions revision b77207bd24ad096ee83b1e1b026d2f130259d616. No benchmark questions were used to train, calibrate, select checkpoints, or tune prompts. Accuracy is higher-is-better; NLL, Brier, and ECE are lower-is-better. ECE is occupancy-weighted, equal-width 15-bin top-label ECE.

0.8B

Model Parameters Accuracy NLL Multiclass Brier 15-bin ECE HF downloads, prior month
Qev 1.0 0.8B 0.8B 32.40% 1.516 0.753 0.10406 0
Kev 0.8B 0.8B Not evaluated on this suite β€” β€” β€” 14,032

2B

Model Parameters Accuracy NLL Multiclass Brier 15-bin ECE HF downloads, prior month
Qev 1.0 2B 2B 31.70% 1.657 0.784 0.15145 0
Decider 2B 2B 39.59% local 1.462 0.736 0.13571 7,391
FLock this-that-model-1.1 1.88B 77.84% local (77.5% card) 0.609 0.303 0.06891 34

4B

Model Parameters Accuracy NLL Multiclass Brier 15-bin ECE HF downloads, prior month
Qev 1.0 4B 4B 42.63% 1.371 0.706 0.11397 0
Kev 4B 4B Not evaluated on this suite β€” β€” β€” 12,499

Decider 2B was run through its native Decider.decide API at bf16 and its stored temperature (2.515). Against it, Qev reaches 80.06% of Decider's accuracy, below the 95% target and 5.92 pp below the 37.61% acceptance threshold. Qev has worse NLL, Brier and corrected ECE. A 50,000-replicate family-cluster bootstrap interval for the accuracy gain is +3.10 to +12.81 pp for Decider; it beat Qev on 14 of 19 families, tied one, and lost four. Qev scores each option separately using its fitted temperature, so serving/scoring differences limit direct calibration and latency comparison. FLock 1.1 used its official single-pass TypedDecider API at bf16 and temperature 1.0; its overlap with this benchmark's rule structures is unverified. Newer FLock 1.2 is excluded because its card lists this exact benchmark as training data.

Metric correction (2026-11-13): An earlier report understated the two local peer ECE values because it omitted bin-occupancy weighting. ECE was recalculated from retained per-item probability vectors using the same occupancy-weighted formula as Qev's evaluator. FLock 1.1 is 0.06891 (supersedes 0.00075), and Decider 2B is 0.13571 (supersedes 0.00109). Qev's values were independently recomputed and are unchanged. Accuracy, NLL, and Brier did not change.

Kev 0.8B and 4B same-suite runs were stopped before scoring any benchmark rows because the pinned official predictor imports POSIX fcntl, unavailable in this Windows runtime; no Kev results are claimed here. Popularity counts are Hugging Face monthly downloads observed 2026-11-13, not unique users or evidence of quality. All comparisons remain single-suite diagnostics; see the Qev comparison report for the other pinned suites, strong popular peers, and their gaps. The multi-suite, independent evaluation and rights gates remain open.

Downloads last month
-
Inference Providers NEW
This model isn't deployed by any Inference Provider. πŸ™‹ Ask for provider support

Model tree for ba-research/Qev1.0-Qwen3.5-4B

Finetuned
Qwen/Qwen3.5-4B
Adapter
(1,112)
this model