Jebadiah 9B v2: model card, merge verification, standalone script

README.md

@@ -0,0 +1,236 @@
+---
+base_model: Qwen/Qwen3.5-9B
+base_model_relation: finetune
+library_name: transformers
+license: apache-2.0
+language:
+- en
+tags:
+- decision-model
+- system-one
+- calibrated-probabilities
+- typed-decisions
+- ainode
+- merged-lora
+datasets:
+- LocalLLaMA/typed-decisions
+- nvidia/HelpSteer2
+- mteb/summeval
+model-index:
+- name: jebadiah-9b-v2
+  results:
+  - task:
+      type: text-classification
+      name: typed decisions (choice, noul, score)
+    dataset:
+      name: Jevals suite 0.1.0, PubMedQA (noul)
+      type: jevals-pubmedqa
+    metrics:
+    - type: accuracy
+      value: 90.3
+    - type: decision_score_jevals
+      value: 68.6
+  - task:
+      type: text-classification
+      name: typed decisions (choice, noul, score)
+    dataset:
+      name: Jevals suite 0.1.0, Banking77 (choice, 77-way)
+      type: jevals-banking77
+    metrics:
+    - type: accuracy
+      value: 70.7
+    - type: decision_score_jevals
+      value: 58.2
+  - task:
+      type: text-classification
+      name: typed decisions (choice, noul, score)
+    dataset:
+      name: Jevals suite 0.1.0, HelpSteer2 helpfulness (score)
+      type: jevals-helpsteer2
+    metrics:
+    - type: decision_score_jevals
+      value: 10.4
+  - task:
+      type: text-classification
+      name: typed decisions (choice, noul, score)
+    dataset:
+      name: Nimble public human-labelled subsets (13, macro accuracy)
+      type: nimble-public
+    metrics:
+    - type: accuracy
+      value: 77.0
+---
+
+# Jebadiah 9B v2
+
+
+Jebadiah (Jeb for short) is Frontier Infra's open System One style decision model, trained and served with AINode: it answers typed questions with a
+probability over the option labels instead of generating text, calibrated per question type when the shipped
+temperatures are applied. Three question types, the
+same three Jev uses: **choice** (pick one of N), **noul** (a yes or no statement, returned as P(yes)) and
+**score** (place the state on an ordered rubric). It is trained with AINode's trainer on public data only
+and served by AINode's `/v1/decide` and `/v1/systemone` routes on any NVIDIA GPU. The routes answer Jev's
+three question types; the request schema is AINode's own (documented on the route), not Jev's exact wire format.
+
+<img src="jeb-banner.png" alt="They call me Jeb. He does not talk much. He just decides." width="100%">
+
+This repository holds **v2 9B** as full bf16 weights: the v2 LoRA merged into `Qwen/Qwen3.5-9B` (pinned revision
+`c2022362`), so it loads with transformers alone, no PEFT and no separate base download.
+
+**What changed from v1 is one thing: the base checkpoint.** v1 was trained on the Base checkpoint
+`Qwen/Qwen3.5-9B-Base` (revision `68c46c4b`). v2 is trained on the **chat** checkpoint `Qwen/Qwen3.5-9B` with thinking off
+(the chat template rendered with `enable_thinking=false`, the prompt contract v1 already used). Everything else
+in the recipe is unchanged from [v1](https://huggingface.co/frontier-infra/jebadiah-9b-v1): the same public pool (11,013 records,
+15,813 questions) and the same train and calibration split, the same LoRA (rank 16, alpha 32, every linear
+projection), the same one epoch, the same learning rate 1e-4 and seed, the same objective (cross-entropy over the
+option-label logits with the ordinal score target) and the same temperature fit. That base swap is what the gain
+is: headline **73.9** against v1's 73.3. The run trained on one NVIDIA A100 80GB PCIe in
+90 minutes. v2 is a fixed version; newer versions are published as new repositories and never
+replace this one.
+
+Made in Texas.
+
+**Read this first.** Served through AINode today, `/v1/systemone` and `/v1/decide` return the model's raw distribution; the per-type temperatures in `temperatures.json` are applied offline in every calibrated number on this card (AINode issue 276 tracks applying them on the route). To reproduce the numbers, or to get the calibrated answer yourself, use the standalone script in `scripts/` (below).
+
+## Results
+
+Measured by us with AINode's bench, one logit read per question, the same rendered prompt for every
+model. Accuracy is the share of questions whose top label is the human label. Decision Score is Jevals'
+metric: 100 = perfect, 0 = guessing the label base rates, below 0 = worse than that. These are our
+numbers on the public suites, not rows on the Jevals board. Headline (macro over the zero-shot public
+sets): **73.9** (v1: 73.3). Its parts: Jevals PubMedQA 90.3 (v1 89.7), Jevals Banking77 70.7 (v1 70.0), Jevals HelpSteer2 40.7 (v1 40.3), Nimble 324 81.2 (v1 78.7), Kev transfer-v4 83.8 (v1 84.0), Nimble public macro 77.0 (v1 77.0).
+
+The v1 columns are the published [v1](https://huggingface.co/frontier-infra/jebadiah-9b-v1) records, same sets, same bench, same repeats.
+
+| Set (questions) | Type | Accuracy | v1 accuracy | Floor (majority label) | Decision Score (Jevals) | v1 Decision Score | ECE, temperatures applied | Flips over identical repeats |
+|---|---|---:|---:|---:|---:|---:|---:|---:|
+| Jevals PubMedQA (300) | noul | 90.3 | 89.7 | 62.0 | 68.6 | 66.8 | 0.025 | 0.0% |
+| Jevals Banking77 (300, 77 options) | choice | 70.7 | 70.0 | 1.3 | 58.2 | 56.9 | 0.076 | 0.7% |
+| Jevals HelpSteer2 helpfulness (300, 5 levels) | score | 40.7 | 40.3 | 41.7 | 10.4 | 11.9 | 0.039 (raw 0.066) | 2.3% |
+| Nimble held-out eval (324) | mixed | 81.2 | 78.7 | 17.6 | 65.0 | 65.0 | 0.085 | 0.3% |
+| Kev transfer-v4 test (764) | mixed | 83.8 | 84.0 | 21.5 | 67.9 | 67.0 | 0.025 | 0.3% |
+| Kev decision-v7 test (1,440) | mixed | 81.2 | 80.8 | 20.3 | 71.4 | 70.6 | 0.050 | n/a |
+| typed-decisions test (2,000) | mixed | 78.3 | 78.9 | 15.3 | 56.5 | 56.4 | 0.156 | 0.1% |
+
+Nimble's 13 public human-labelled subsets (3,880 questions), macro accuracy: **77.0** (v1: 77.0).
+For scale, Bespoke's published table puts Nimble-9B at 74.8 and Jev at 76.0 on the same subsets with their scorer.
+
+| Nimble public subset (questions) | Accuracy | v1 accuracy | Decision Score (Jevals) | v1 Decision Score |
+|---|---:|---:|---:|---:|
+| aegis2 (250) | 78.4 | 79.2 | 38.0 | 35.8 |
+| boolq (300) | 86.0 | 85.7 | 54.6 | 57.7 |
+| civil_comments (300) | 86.0 | 86.7 | -7.4 | -4.8 |
+| helpsteer2 (249) | 43.4 | 41.0 | 13.3 | 15.2 |
+| massive-de-DE (350) | 84.6 | 84.9 | 74.3 | 73.3 |
+| massive-en-US (350) | 85.4 | 85.7 | 77.3 | 76.0 |
+| multinli (299) | 84.9 | 87.6 | 67.2 | 70.6 |
+| paws (250) | 83.6 | 85.6 | 54.0 | 57.2 |
+| pubmedqa (250) | 72.8 | 68.8 | 38.5 | 34.5 |
+| squad2 (299) | 79.9 | 80.3 | 42.9 | 43.6 |
+| summeval-consistency (144) | 87.5 | 87.5 | 42.6 | 36.0 |
+| summeval-relevance (240) | 52.5 | 51.7 | 23.4 | 25.9 |
+| vitaminc-dev (599) | 75.6 | 76.3 | 41.8 | 39.0 |
+
+Where v2 is not better. Jevals HelpSteer2 Decision Score is 10.4 against v1's 11.9, and its top level flips over identical repeats on 2.3% of questions against 0.3%. The Nimble public macro is flat (77.0 against 77.0): MultiNLI (84.9 against 87.6) and PAWS (83.6 against 85.6) give back what PubMedQA (72.8 against 68.8) and HelpSteer2 gain. ECE on the Nimble 324 set rises from 0.063 to 0.085. The headline gain is 0.6 points, most of it from the Nimble 324 set (81.2 against 78.7).
+
+Two things to read carefully, unchanged from v1 because the pool is unchanged. HelpSteer2 and SummEval are not
+zero-shot for Jeb: the pool trains on the HelpSteer2 **train** split (the Jevals and Nimble items come from
+validation, and no item overlaps) and on SummEval articles that Nimble does not score (its 15 evaluation articles
+are held out entirely), so those rows are held-out items of a seen rubric. And the in-distribution sets
+(typed-decisions, Kev decision-v7) are the test splits of sources in the pool.
+
+Every eval record (per question: option keys, probabilities, pick, label, repeat, option order) is in
+`eval/`, with `eval/results.json` carrying the full metric set. The records were scored on the unmerged adapter;
+the merged weights here reproduce them (see Training, merge).
+
+## Robustness to irrelevant content (nonce test)
+
+The nonce pass (a fresh UUID planted in the state or appended to the instructions, three passes per question on the four head sets, as on the v1 card) had not finished for this model when this card was written. It is pending; no robustness number is claimed here until its records exist.
+
+## How it decides
+
+**Which temperature fit, and why.** `temperatures.json` carries two fits on the calibration split (689 records, 921 questions: choice 225, noul 147, score 549). The `hard` fit minimises NLL against the argmax label and sharpens (T 0.79 / 0.67 / 0.83). The `train` fit minimises NLL against the training target, the soft or ordinal distribution the model was taught, and softens (T 1.19 / 1.09 / 1.22). v2 applies `train`, as v1 does, because the ordinal target is what makes rubric probabilities honest, and the `hard` fit is exactly the sharpening that made v0's HelpSteer2 calibration worse. The price is visible in the same file: on the calibration split measured against hard labels, ECE moves from 0.074 to 0.089 (choice), 0.064 to 0.064 (noul) and 0.092 to 0.128 (score) while NLL on the training target falls (0.4594 to 0.4542, 0.3776 to 0.3768, 1.1017 to 1.0929). The ECE column in the results table is measured on the evaluation sets, not on this split, with the applied temperatures. Both fits are in the file; a consumer who gates on the argmax may prefer `hard`, and either way should refit on its own data.
+
+The prompt is AINode's own decide rendering (`ainode.api.decide.build_messages`, source commit
+`e5c08938`, `prompt_source_sha256` in `prompt_contract.json`), through the chat template with
+thinking off. The option labels are single tokens; the model's answer is the distribution over those label
+tokens at the last prompt position, read in fp32 from the last hidden state and then temperature scaled
+per type (`temperatures.json`: choice 1.19, noul 1.09, score 1.22). Nothing is generated.
+
+The supported way to run it is AINode, which renders the prompt exactly as trained (a worked
+`/v1/systemone` request and response is in the AINode docs issue tracker until the page lands):
+
+```bash
+curl -sS https://<your-ainode>/v1/systemone -H "Authorization: Bearer <key>" -H "Content-Type: application/json" -d '{
+  "model": "frontier-infra/jebadiah-9b-v2",
+  "state": {"ticket": "Customer says the invoice total does not match the quote."},
+  "questions": {
+    "route": {"type": "choice", "instructions": "Which team should take this ticket?", "criteria": {"billing": "an invoice, a charge or a refund", "support": "a product question", "sales": "a quote or a renewal"}},
+    "urgent": {"type": "noul", "instructions": "The customer is blocked from working.", "criteria": {"true": "work has stopped", "false": "it can wait"}}
+  }
+}'
+```
+
+The weights keep the base repository's exact layout (`Qwen3_5ForConditionalGeneration`, the same `config.json`,
+tokenizer and chat template as `Qwen/Qwen3.5-9B` at `c2022362`); only the LoRA-targeted language-model weights differ.
+The vision tower is the base's, untouched and unused.
+
+### Standalone, reproducible
+
+`scripts/` in this repository carries the trainer's own copies of the renderer and the logit read: `ainode_prompt_verbatim.py` is AINode's decide rendering copied verbatim from [getainode/ainode at e5c08938](https://github.com/getainode/ainode/blob/e5c089386e0239c9eb270eeb490d181722b8da5b/ainode/api/decide.py) (the sha256 of its copied definitions is the `prompt_source_sha256` in `prompt_contract.json`), `jebadiah_prompt.py` turns a `/v1/systemone` question into that prompt with single-token labels, `jebadiah_model.py` reads the label-token logits at the answer position in fp32, and `decide_standalone.py` runs one request end to end. These four files are byte-identical to v1's except `decide_standalone.py`, which takes the merged model directory instead of a base plus an adapter:
+
+```bash
+pip install torch transformers==5.17.0 huggingface_hub
+huggingface-cli download frontier-infra/jebadiah-9b-v2 --local-dir jeb
+python jeb/scripts/decide_standalone.py --model jeb --request jeb/scripts/example-request.json   # add --no-temperatures for the raw distribution
+```
+
+It prints the answers in the route's shape (choice: `choice`, `confidence`, `probabilities`; noul: `noul` = P(true); score: `score`, `legend`, `probabilities`) plus the temperatures it applied. Runs on CPU too (slow without the DeltaNet kernels), and `--device mps` on Apple silicon, which is how it was checked before publishing.
+
+Thresholds belong to the caller: act on a high probability, confirm or escalate on a middle one, hand a
+low one to a person or a bigger model. The model never refuses; policy is built from decisions.
+
+## Training
+
+- **Base:** `Qwen/Qwen3.5-9B` at revision `c202236235762e1c871ad0ccb60c8ee5ba337b9a`, the chat checkpoint, prompted with thinking off. (v1: `Qwen/Qwen3.5-9B-Base`.)
+- **Objective:** cross-entropy over the candidate option-label logits at the answer position. Choice and
+  noul use the source's gold distribution where it provides one (typed-decisions) and the hard label
+  otherwise; score questions use an ordinal kernel around the human label (adjacent level 0.2 of the
+  label's weight), and the score temperature is fitted on that same target. Unchanged from v1.
+- **Adapter, before merging:** LoRA r=16, alpha=32, dropout 0.05 on every linear projection including the Gated DeltaNet
+  projections, 43.3M trainable parameters. bf16, SDPA attention, micro-batch 8, learning rate 1e-4,
+  one epoch, 1,862 steps, gradient checkpointing, seed 17. Unchanged from v1.
+- **Data (public only), 11,013 records and 15,813 questions in the pool, unchanged from v1:** `LocalLLaMA/typed-decisions` train (Apache-2.0,
+  revision `ea930645`); the Kev v7 training sources whose licenses permit derived weights (BoolQ, MNLI,
+  DBpedia14 and Kev's contrastive and composition sets, Kev suites revision `a88f56db`); `nvidia/HelpSteer2`
+  train split (CC BY 4.0, 2,573 score questions: helpfulness in the Jevals task's exact shape for 1,609
+  rows, plus correctness, coherence, complexity and verbosity on 241 of them); SummEval (MIT, via
+  `mteb/summeval`, 1,664 score questions over four dimensions on 26 articles, with Nimble's 15 evaluation
+  articles excluded). 95/5 family split into 10,324 training records (14,892 training questions) and 689 calibration records (921 calibration questions).
+  Nimble's train set is excluded (no license stated); the Jevals test items are excluded from training.
+  No private data of any kind.
+- **Compute:** one NVIDIA A100 80GB PCIe, 90 min training, 16 min evaluation, torch 2.11, PEFT 0.21. (v1 trained on an H100 PCIe.)
+- **Merge:** W' = W + (alpha / r) B A = W + 2.0 B A on every LoRA-targeted weight, done shard by shard on the CPU
+  from the safetensors files in fp32 and stored in bf16: 248 of 248 LoRA pairs merged, none unmatched, in 35 s.
+  Every other tensor is the base's, bit for bit. Checked after merging: the merged weights load with no missing or
+  unexpected keys, and re-scoring 260 test questions (Jevals PubMedQA, Banking77 and HelpSteer2, Nimble 324;
+  repeat 0 in the run's own option order, temperatures applied) on Apple MPS in bf16 gives the same pick as the run's
+  records on **257 of 260** (choice 83/84, noul 89/89, score 85/87); the 3 that differ were near ties in the run's own record (top-two margin at most 0.0106), and 1 of them already changed pick between the run's own repeats. The largest probability difference on any
+  option is 0.0291 (median 0.0021), which is bf16 across two devices
+  (CUDA with the unmerged adapter for the records, MPS with merged weights here). Verification records: `eval/merge-verify.json`.
+
+## Limitations
+
+- Single-hop judgments only. Split a chain of inference into hops.
+- AINode caps a choice question at 20 options; Banking77's 77 options were scored with an extended
+  single-token alphabet for the benchmark only.
+- English data, plus Massive's German subset scoring well by accident of the base model.
+- Calibration was fitted on the training distribution. Refit the temperatures on your own data before
+  trusting a threshold, and remember the served route applies none yet.
+
+## Versioning and license
+
+v2 is frozen. Later versions land as `jebadiah-<size>-v<N>` repositories; [v1](https://huggingface.co/frontier-infra/jebadiah-9b-v1) stays
+as it is. The weights are Apache-2.0, the base model `Qwen/Qwen3.5-9B` is Apache-2.0. Evaluation data: Jevals suite 0.1.0 (CC-BY-4.0, "Jevals (jevals.com),
+release 2026-09-18"), Nimble public subsets (Bespoke Labs), Kev test sets and typed-decisions test,
+each under its own license.

eval/merge-verify.json

@@ -0,0 +1,2700 @@
+{
+ "summary": {
+  "merged": "jebadiah-9b-v2",
+  "run": "9b-chat-v1",
+  "device": "mps",
+  "dtype": "bf16 weights, fp32 candidate head",
+  "temperatures": {
+   "choice": 1.1863,
+   "noul": 1.0903,
+   "score": 1.2162
+  },
+  "questions": 260,
+  "identical_picks": 257,
+  "by_type": {
+   "choice": {
+    "identical": 83,
+    "questions": 84
+   },
+   "noul": {
+    "identical": 89,
+    "questions": 89
+   },
+   "score": {
+    "identical": 85,
+    "questions": 87
+   }
+  },
+  "by_set": {
+   "jevals-pubmedqa": 80,
+   "jevals-banking77": 70,
+   "jevals-helpsteer2": 80,
+   "nimble-eval": 30
+  },
+  "max_abs_prob_diff": {
+   "median": 0.002068277301788324,
+   "p95": 0.012686089736938522,
+   "max": 0.029109271339416543
+  },
+  "differing": [
+   {
+    "set": "jevals-banking77",
+    "id": "jevals-banking77:banking77-882",
+    "qid": "decision",
+    "type": "choice",
+    "pick": "supported_cards_and_currencies",
+    "pick_run": "fiat_currency_support",
+    "same": false,
+    "max_abs_dp": 0.01041107138633729,
+    "run_top2_margin": 0.010566,
+    "run_picks_over_repeats": [
+     "fiat_currency_support",
+     "fiat_currency_support"
+    ]
+   },
+   {
+    "set": "jevals-helpsteer2",
+    "id": "jevals-helpsteer2:helpsteer2-972",
+    "qid": "decision",
+    "type": "score",
+    "pick": "3",
+    "pick_run": "2",
+    "same": false,
+    "max_abs_dp": 0.007170431488037116,
+    "run_top2_margin": 0.00078,
+    "run_picks_over_repeats": [
+     "2",
+     "2",
+     "2",
+     "2",
+     "2"
+    ]
+   },
+   {
+    "set": "jevals-helpsteer2",
+    "id": "jevals-helpsteer2:helpsteer2-945",
+    "qid": "decision",
+    "type": "score",
+    "pick": "2",
+    "pick_run": "3",
+    "same": false,
+    "max_abs_dp": 0.003689604001998914,
+    "run_top2_margin": 0.000376,
+    "run_picks_over_repeats": [
+     "3",
+     "3",
+     "3",
+     "3",
+     "2"
+    ]
+   }
+  ],
+  "load_s": 42.8,
+  "total_s": 702.9,
+  "reference": "the run's eval-adapter rows, repeat 0, order seed 0 (CUDA A100, bf16, PEFT adapter unmerged)",
+  "differing_max_run_margin": 0.010566,
+  "differing_flipped_in_run_repeats": 1
+ },
+ "rows": [
+  {
+   "set": "jevals-pubmedqa",
+   "id": "jevals-pubmedqa:pubmedqa-577",
+   "qid": "decision",
+   "type": "noul",
+   "pick": "false",
+   "pick_run": "false",
+   "same": true,
+   "max_abs_dp": 0.002361614906311038
+  },
+  {
+   "set": "jevals-pubmedqa",
+   "id": "jevals-pubmedqa:pubmedqa-229",
+   "qid": "decision",
+   "type": "noul",
+   "pick": "false",
+   "pick_run": "false",
+   "same": true,
+   "max_abs_dp": 0.0007400083885192865
+  },
+  {
+   "set": "jevals-pubmedqa",
+   "id": "jevals-pubmedqa:pubmedqa-874",
+   "qid": "decision",
+   "type": "noul",
+   "pick": "true",
+   "pick_run": "true",
+   "same": true,
+   "max_abs_dp": 0.0007248036618232728
+  },
+  {
+   "set": "jevals-pubmedqa",
+   "id": "jevals-pubmedqa:pubmedqa-214",
+   "qid": "decision",
+   "type": "noul",
+   "pick": "true",
+   "pick_run": "true",
+   "same": true,
+   "max_abs_dp": 8.721527481081459e-05
+  },
+  {
+   "set": "jevals-pubmedqa",
+   "id": "jevals-pubmedqa:pubmedqa-758",
+   "qid": "decision",
+   "type": "noul",
+   "pick": "true",
+   "pick_run": "true",
+   "same": true,
+   "max_abs_dp": 0.00020452411460876452
+  },
+  {
+   "set": "jevals-pubmedqa",
+   "id": "jevals-pubmedqa:pubmedqa-967",
+   "qid": "decision",
+   "type": "noul",
+   "pick": "false",
+   "pick_run": "false",
+   "same": true,
+   "max_abs_dp": 0.0011608515167236266
+  },
+  {
+   "set": "jevals-pubmedqa",
+   "id": "jevals-pubmedqa:pubmedqa-405",
+   "qid": "decision",
+   "type": "noul",
+   "pick": "true",
+   "pick_run": "true",
+   "same": true,
+   "max_abs_dp": 0.000412434085845903
+  },
+  {
+   "set": "jevals-pubmedqa",
+   "id": "jevals-pubmedqa:pubmedqa-970",
+   "qid": "decision",
+   "type": "noul",
+   "pick": "true",
+   "pick_run": "true",
+   "same": true,
+   "max_abs_dp": 4.357640852034085e-05
+  },
+  {
+   "set": "jevals-pubmedqa",
+   "id": "jevals-pubmedqa:pubmedqa-842",
+   "qid": "decision",
+   "type": "noul",
+   "pick": "true",
+   "pick_run": "true",
+   "same": true,
+   "max_abs_dp": 0.00011573467636105672
+  },
+  {
+   "set": "jevals-pubmedqa",
+   "id": "jevals-pubmedqa:pubmedqa-697",
+   "qid": "decision",
+   "type": "noul",
+   "pick": "true",
+   "pick_run": "true",
+   "same": true,
+   "max_abs_dp": 0.002668783897876739
+  },
+  {
+   "set": "jevals-pubmedqa",
+   "id": "jevals-pubmedqa:pubmedqa-130",
+   "qid": "decision",
+   "type": "noul",
+   "pick": "true",
+   "pick_run": "true",
+   "same": true,
+   "max_abs_dp": 0.0007475724792480598
+  },
+  {
+   "set": "jevals-pubmedqa",
+   "id": "jevals-pubmedqa:pubmedqa-880",
+   "qid": "decision",
+   "type": "noul",
+   "pick": "true",
+   "pick_run": "true",
+   "same": true,
+   "max_abs_dp": 0.0019318389310836737
+  },
+  {
+   "set": "jevals-pubmedqa",
+   "id": "jevals-pubmedqa:pubmedqa-463",
+   "qid": "decision",
+   "type": "noul",
+   "pick": "true",
+   "pick_run": "true",
+   "same": true,
+   "max_abs_dp": 0.0015171572542190537
+  },
+  {
+   "set": "jevals-pubmedqa",
+   "id": "jevals-pubmedqa:pubmedqa-16",
+   "qid": "decision",
+   "type": "noul",
+   "pick": "true",
+   "pick_run": "true",
+   "same": true,
+   "max_abs_dp": 0.008817474800109859
+  },
+  {
+   "set": "jevals-pubmedqa",
+   "id": "jevals-pubmedqa:pubmedqa-531",
+   "qid": "decision",
+   "type": "noul",
+   "pick": "true",
+   "pick_run": "true",
+   "same": true,
+   "max_abs_dp": 0.0028381280999183606
+  },
+  {
+   "set": "jevals-pubmedqa",
+   "id": "jevals-pubmedqa:pubmedqa-604",
+   "qid": "decision",
+   "type": "noul",
+   "pick": "false",
+   "pick_run": "false",
+   "same": true,
+   "max_abs_dp": 0.004145436416626014
+  },
+  {
+   "set": "jevals-pubmedqa",
+   "id": "jevals-pubmedqa:pubmedqa-409",
+   "qid": "decision",
+   "type": "noul",
+   "pick": "true",
+   "pick_run": "true",
+   "same": true,
+   "max_abs_dp": 0.0076537508029937695
+  },
+  {
+   "set": "jevals-pubmedqa",
+   "id": "jevals-pubmedqa:pubmedqa-388",
+   "qid": "decision",
+   "type": "noul",
+   "pick": "true",
+   "pick_run": "true",
+   "same": true,
+   "max_abs_dp": 3.363730621341876e-05
+  },
+  {
+   "set": "jevals-pubmedqa",
+   "id": "jevals-pubmedqa:pubmedqa-811",
+   "qid": "decision",
+   "type": "noul",
+   "pick": "true",
+   "pick_run": "true",
+   "same": true,
+   "max_abs_dp": 0.002041696565628026
+  },
+  {
+   "set": "jevals-pubmedqa",
+   "id": "jevals-pubmedqa:pubmedqa-991",
+   "qid": "decision",
+   "type": "noul",
+   "pick": "true",
+   "pick_run": "true",
+   "same": true,
+   "max_abs_dp": 0.00605180963134766
+  },
+  {
+   "set": "jevals-pubmedqa",
+   "id": "jevals-pubmedqa:pubmedqa-11",
+   "qid": "decision",
+   "type": "noul",
+   "pick": "false",
+   "pick_run": "false",
+   "same": true,
+   "max_abs_dp": 0.003769225850105279
+  },
+  {
+   "set": "jevals-pubmedqa",
+   "id": "jevals-pubmedqa:pubmedqa-192",
+   "qid": "decision",
+   "type": "noul",
+   "pick": "true",
+   "pick_run": "true",
+   "same": true,
+   "max_abs_dp": 0.004384308416366556
+  },
+  {
+   "set": "jevals-pubmedqa",
+   "id": "jevals-pubmedqa:pubmedqa-337",
+   "qid": "decision",
+   "type": "noul",
+   "pick": "true",
+   "pick_run": "true",
+   "same": true,
+   "max_abs_dp": 0.011962067066192628
+  },
+  {
+   "set": "jevals-pubmedqa",
+   "id": "jevals-pubmedqa:pubmedqa-867",
+   "qid": "decision",
+   "type": "noul",
+   "pick": "false",
+   "pick_run": "false",
+   "same": true,
+   "max_abs_dp": 0.000688766813755036
+  },
+  {
+   "set": "jevals-pubmedqa",
+   "id": "jevals-pubmedqa:pubmedqa-441",
+   "qid": "decision",
+   "type": "noul",
+   "pick": "false",
+   "pick_run": "false",
+   "same": true,
+   "max_abs_dp": 0.0005272247047424239
+  },
+  {
+   "set": "jevals-pubmedqa",
+   "id": "jevals-pubmedqa:pubmedqa-980",
+   "qid": "decision",
+   "type": "noul",
+   "pick": "true",
+   "pick_run": "true",
+   "same": true,
+   "max_abs_dp": 0.0003238794937133749
+  },
+  {
+   "set": "jevals-pubmedqa",
+   "id": "jevals-pubmedqa:pubmedqa-736",
+   "qid": "decision",
+   "type": "noul",
+   "pick": "true",
+   "pick_run": "true",
+   "same": true,
+   "max_abs_dp": 0.0025702828493118224
+  },
+  {
+   "set": "jevals-pubmedqa",
+   "id": "jevals-pubmedqa:pubmedqa-569",
+   "qid": "decision",
+   "type": "noul",
+   "pick": "true",
+   "pick_run": "true",
+   "same": true,
+   "max_abs_dp": 0.00014876399350166256
+  },
+  {
+   "set": "jevals-pubmedqa",
+   "id": "jevals-pubmedqa:pubmedqa-359",
+   "qid": "decision",
+   "type": "noul",
+   "pick": "false",
+   "pick_run": "false",
+   "same": true,
+   "max_abs_dp": 0.0013036476001739433
+  },
+  {
+   "set": "jevals-pubmedqa",
+   "id": "jevals-pubmedqa:pubmedqa-401",
+   "qid": "decision",
+   "type": "noul",
+   "pick": "true",
+   "pick_run": "true",
+   "same": true,
+   "max_abs_dp": 0.0001761184187531479
+  },
+  {
+   "set": "jevals-pubmedqa",
+   "id": "jevals-pubmedqa:pubmedqa-754",
+   "qid": "decision",
+   "type": "noul",
+   "pick": "false",
+   "pick_run": "false",
+   "same": true,
+   "max_abs_dp": 0.003703943653106667
+  },
+  {
+   "set": "jevals-pubmedqa",
+   "id": "jevals-pubmedqa:pubmedqa-858",
+   "qid": "decision",
+   "type": "noul",
+   "pick": "false",
+   "pick_run": "false",
+   "same": true,
+   "max_abs_dp": 0.001998045045375818
+  },
+  {
+   "set": "jevals-pubmedqa",
+   "id": "jevals-pubmedqa:pubmedqa-789",
+   "qid": "decision",
+   "type": "noul",
+   "pick": "false",
+   "pick_run": "false",
+   "same": true,
+   "max_abs_dp": 7.641987609863232e-05
+  },
+  {
+   "set": "jevals-pubmedqa",
+   "id": "jevals-pubmedqa:pubmedqa-52",
+   "qid": "decision",
+   "type": "noul",
+   "pick": "false",
+   "pick_run": "false",
+   "same": true,
+   "max_abs_dp": 0.004233029576301561
+  },
+  {
+   "set": "jevals-pubmedqa",
+   "id": "jevals-pubmedqa:pubmedqa-119",
+   "qid": "decision",
+   "type": "noul",
+   "pick": "true",
+   "pick_run": "true",
+   "same": true,
+   "max_abs_dp": 0.00014103620529171135
+  },
+  {
+   "set": "jevals-pubmedqa",
+   "id": "jevals-pubmedqa:pubmedqa-148",
+   "qid": "decision",
+   "type": "noul",
+   "pick": "true",
+   "pick_run": "true",
+   "same": true,
+   "max_abs_dp": 3.7405700683557086e-06
+  },
+  {
+   "set": "jevals-pubmedqa",
+   "id": "jevals-pubmedqa:pubmedqa-519",
+   "qid": "decision",
+   "type": "noul",
+   "pick": "true",
+   "pick_run": "true",
+   "same": true,
+   "max_abs_dp": 3.914723384380349e-05
+  },
+  {
+   "set": "jevals-pubmedqa",
+   "id": "jevals-pubmedqa:pubmedqa-812",
+   "qid": "decision",
+   "type": "noul",
+   "pick": "false",
+   "pick_run": "false",
+   "same": true,
+   "max_abs_dp": 1.720184409618325e-05
+  },
+  {
+   "set": "jevals-pubmedqa",
+   "id": "jevals-pubmedqa:pubmedqa-276",
+   "qid": "decision",
+   "type": "noul",
+   "pick": "true",
+   "pick_run": "true",
+   "same": true,
+   "max_abs_dp": 0.001692070934534079
+  },
+  {
+   "set": "jevals-pubmedqa",
+   "id": "jevals-pubmedqa:pubmedqa-505",
+   "qid": "decision",
+   "type": "noul",
+   "pick": "false",
+   "pick_run": "false",
+   "same": true,
+   "max_abs_dp": 0.029109271339416543
+  },
+  {
+   "set": "jevals-pubmedqa",
+   "id": "jevals-pubmedqa:pubmedqa-333",
+   "qid": "decision",
+   "type": "noul",
+   "pick": "true",
+   "pick_run": "true",
+   "same": true,
+   "max_abs_dp": 0.01489011363983156
+  },
+  {
+   "set": "jevals-pubmedqa",
+   "id": "jevals-pubmedqa:pubmedqa-746",
+   "qid": "decision",
+   "type": "noul",
+   "pick": "true",
+   "pick_run": "true",
+   "same": true,
+   "max_abs_dp": 0.00012200270804762874
+  },
+  {
+   "set": "jevals-pubmedqa",
+   "id": "jevals-pubmedqa:pubmedqa-644",
+   "qid": "decision",
+   "type": "noul",
+   "pick": "false",
+   "pick_run": "false",
+   "same": true,
+   "max_abs_dp": 9.002473068242534e-05
+  },
+  {
+   "set": "jevals-pubmedqa",
+   "id": "jevals-pubmedqa:pubmedqa-248",
+   "qid": "decision",
+   "type": "noul",
+   "pick": "true",
+   "pick_run": "true",
+   "same": true,
+   "max_abs_dp": 0.00016415084588527681
+  },
+  {
+   "set": "jevals-pubmedqa",
+   "id": "jevals-pubmedqa:pubmedqa-764",
+   "qid": "decision",
+   "type": "noul",
+   "pick": "false",
+   "pick_run": "false",
+   "same": true,
+   "max_abs_dp": 0.00045313702392578425
+  },
+  {
+   "set": "jevals-pubmedqa",
+   "id": "jevals-pubmedqa:pubmedqa-711",
+   "qid": "decision",
+   "type": "noul",
+   "pick": "true",
+   "pick_run": "true",
+   "same": true,
+   "max_abs_dp": 0.00029391733694076466
+  },
+  {
+   "set": "jevals-pubmedqa",
+   "id": "jevals-pubmedqa:pubmedqa-941",
+   "qid": "decision",
+   "type": "noul",
+   "pick": "true",
+   "pick_run": "true",
+   "same": true,
+   "max_abs_dp": 0.0019045123214721515
+  },
+  {
+   "set": "jevals-pubmedqa",
+   "id": "jevals-pubmedqa:pubmedqa-651",
+   "qid": "decision",
+   "type": "noul",
+   "pick": "false",
+   "pick_run": "false",
+   "same": true,
+   "max_abs_dp": 0.0021859156303405958
+  },
+  {
+   "set": "jevals-pubmedqa",
+   "id": "jevals-pubmedqa:pubmedqa-846",
+   "qid": "decision",
+   "type": "noul",
+   "pick": "true",
+   "pick_run": "true",
+   "same": true,
+   "max_abs_dp": 0.0003319750289917378
+  },
+  {
+   "set": "jevals-pubmedqa",
+   "id": "jevals-pubmedqa:pubmedqa-906",
+   "qid": "decision",
+   "type": "noul",
+   "pick": "false",
+   "pick_run": "false",
+   "same": true,
+   "max_abs_dp": 0.00033138166332245117
+  },
+  {
+   "set": "jevals-pubmedqa",
+   "id": "jevals-pubmedqa:pubmedqa-931",
+   "qid": "decision",
+   "type": "noul",
+   "pick": "false",
+   "pick_run": "false",
+   "same": true,
+   "max_abs_dp": 0.013560935886383074
+  },
+  {
+   "set": "jevals-pubmedqa",
+   "id": "jevals-pubmedqa:pubmedqa-74",
+   "qid": "decision",
+   "type": "noul",
+   "pick": "true",
+   "pick_run": "true",
+   "same": true,
+   "max_abs_dp": 0.000264806226700544
+  },
+  {
+   "set": "jevals-pubmedqa",
+   "id": "jevals-pubmedqa:pubmedqa-58",
+   "qid": "decision",
+   "type": "noul",
+   "pick": "true",
+   "pick_run": "true",
+   "same": true,
+   "max_abs_dp": 0.006453835803985619
+  },
+  {
+   "set": "jevals-pubmedqa",
+   "id": "jevals-pubmedqa:pubmedqa-878",
+   "qid": "decision",
+   "type": "noul",
+   "pick": "true",
+   "pick_run": "true",
+   "same": true,
+   "max_abs_dp": 3.280313563346765e-05
+  },
+  {
+   "set": "jevals-pubmedqa",
+   "id": "jevals-pubmedqa:pubmedqa-217",
+   "qid": "decision",
+   "type": "noul",
+   "pick": "false",
+   "pick_run": "false",
+   "same": true,
+   "max_abs_dp": 0.007298624530792219
+  },
+  {
+   "set": "jevals-pubmedqa",
+   "id": "jevals-pubmedqa:pubmedqa-966",
+   "qid": "decision",
+   "type": "noul",
+   "pick": "false",
+   "pick_run": "false",
+   "same": true,
+   "max_abs_dp": 0.0016663671798705648
+  },
+  {
+   "set": "jevals-pubmedqa",
+   "id": "jevals-pubmedqa:pubmedqa-304",
+   "qid": "decision",
+   "type": "noul",
+   "pick": "true",
+   "pick_run": "true",
+   "same": true,
+   "max_abs_dp": 0.0006484546394348589
+  },
+  {
+   "set": "jevals-pubmedqa",
+   "id": "jevals-pubmedqa:pubmedqa-60",
+   "qid": "decision",
+   "type": "noul",
+   "pick": "true",
+   "pick_run": "true",
+   "same": true,
+   "max_abs_dp": 0.00080796931374073
+  },
+  {
+   "set": "jevals-pubmedqa",
+   "id": "jevals-pubmedqa:pubmedqa-624",
+   "qid": "decision",
+   "type": "noul",
+   "pick": "true",
+   "pick_run": "true",
+   "same": true,
+   "max_abs_dp": 3.0353588104214957e-05
+  },
+  {
+   "set": "jevals-pubmedqa",
+   "id": "jevals-pubmedqa:pubmedqa-547",
+   "qid": "decision",
+   "type": "noul",
+   "pick": "true",
+   "pick_run": "true",
+   "same": true,
+   "max_abs_dp": 0.0085917929992676
+  },
+  {
+   "set": "jevals-pubmedqa",
+   "id": "jevals-pubmedqa:pubmedqa-374",
+   "qid": "decision",
+   "type": "noul",
+   "pick": "false",
+   "pick_run": "false",
+   "same": true,
+   "max_abs_dp": 0.0004448536033629935
+  },
+  {
+   "set": "jevals-pubmedqa",
+   "id": "jevals-pubmedqa:pubmedqa-774",
+   "qid": "decision",
+   "type": "noul",
+   "pick": "true",
+   "pick_run": "true",
+   "same": true,
+   "max_abs_dp": 0.00010372769165034601
+  },
+  {
+   "set": "jevals-pubmedqa",
+   "id": "jevals-pubmedqa:pubmedqa-872",
+   "qid": "decision",
+   "type": "noul",
+   "pick": "false",
+   "pick_run": "false",
+   "same": true,
+   "max_abs_dp": 0.002048760414123524
+  },
+  {
+   "set": "jevals-pubmedqa",
+   "id": "jevals-pubmedqa:pubmedqa-524",
+   "qid": "decision",
+   "type": "noul",
+   "pick": "true",
+   "pick_run": "true",
+   "same": true,
+   "max_abs_dp": 0.0004191811428070069
+  },
+  {
+   "set": "jevals-pubmedqa",
+   "id": "jevals-pubmedqa:pubmedqa-404",
+   "qid": "decision",
+   "type": "noul",
+   "pick": "false",
+   "pick_run": "false",
+   "same": true,
+   "max_abs_dp": 0.00030116800451278694
+  },
+  {
+   "set": "jevals-pubmedqa",
+   "id": "jevals-pubmedqa:pubmedqa-829",
+   "qid": "decision",
+   "type": "noul",
+   "pick": "false",
+   "pick_run": "false",
+   "same": true,
+   "max_abs_dp": 7.498416900619809e-06
+  },
+  {
+   "set": "jevals-pubmedqa",
+   "id": "jevals-pubmedqa:pubmedqa-386",
+   "qid": "decision",
+   "type": "noul",
+   "pick": "false",
+   "pick_run": "false",
+   "same": true,
+   "max_abs_dp": 0.0005611235961914263
+  },
+  {
+   "set": "jevals-pubmedqa",
+   "id": "jevals-pubmedqa:pubmedqa-675",
+   "qid": "decision",
+   "type": "noul",
+   "pick": "true",
+   "pick_run": "true",
+   "same": true,
+   "max_abs_dp": 0.00023019430702924705
+  },
+  {
+   "set": "jevals-pubmedqa",
+   "id": "jevals-pubmedqa:pubmedqa-555",
+   "qid": "decision",
+   "type": "noul",
+   "pick": "true",
+   "pick_run": "true",
+   "same": true,
+   "max_abs_dp": 4.606273269658345e-05
+  },
+  {
+   "set": "jevals-pubmedqa",
+   "id": "jevals-pubmedqa:pubmedqa-963",
+   "qid": "decision",
+   "type": "noul",
+   "pick": "true",
+   "pick_run": "true",
+   "same": true,
+   "max_abs_dp": 0.0004458992661237719
+  },
+  {
+   "set": "jevals-pubmedqa",
+   "id": "jevals-pubmedqa:pubmedqa-314",
+   "qid": "decision",
+   "type": "noul",
+   "pick": "true",
+   "pick_run": "true",
+   "same": true,
+   "max_abs_dp": 0.00030999475097659435
+  },
+  {
+   "set": "jevals-pubmedqa",
+   "id": "jevals-pubmedqa:pubmedqa-895",
+   "qid": "decision",
+   "type": "noul",
+   "pick": "false",
+   "pick_run": "false",
+   "same": true,
+   "max_abs_dp": 0.004996557075500507
+  },
+  {
+   "set": "jevals-pubmedqa",
+   "id": "jevals-pubmedqa:pubmedqa-342",
+   "qid": "decision",
+   "type": "noul",
+   "pick": "false",
+   "pick_run": "false",
+   "same": true,
+   "max_abs_dp": 0.006861803737640337
+  },
+  {
+   "set": "jevals-pubmedqa",
+   "id": "jevals-pubmedqa:pubmedqa-656",
+   "qid": "decision",
+   "type": "noul",
+   "pick": "false",
+   "pick_run": "false",
+   "same": true,
+   "max_abs_dp": 0.0004893547134399157
+  },
+  {
+   "set": "jevals-pubmedqa",
+   "id": "jevals-pubmedqa:pubmedqa-79",
+   "qid": "decision",
+   "type": "noul",
+   "pick": "true",
+   "pick_run": "true",
+   "same": true,
+   "max_abs_dp": 2.925255966190754e-05
+  },
+  {
+   "set": "jevals-pubmedqa",
+   "id": "jevals-pubmedqa:pubmedqa-715",
+   "qid": "decision",
+   "type": "noul",
+   "pick": "false",
+   "pick_run": "false",
+   "same": true,
+   "max_abs_dp": 0.0007816578140258379
+  },
+  {
+   "set": "jevals-pubmedqa",
+   "id": "jevals-pubmedqa:pubmedqa-368",
+   "qid": "decision",
+   "type": "noul",
+   "pick": "false",
+   "pick_run": "false",
+   "same": true,
+   "max_abs_dp": 0.0002648297873735425
+  },
+  {
+   "set": "jevals-pubmedqa",
+   "id": "jevals-pubmedqa:pubmedqa-226",
+   "qid": "decision",
+   "type": "noul",
+   "pick": "true",
+   "pick_run": "true",
+   "same": true,
+   "max_abs_dp": 0.000312848361968987
+  },
+  {
+   "set": "jevals-pubmedqa",
+   "id": "jevals-pubmedqa:pubmedqa-743",
+   "qid": "decision",
+   "type": "noul",
+   "pick": "false",
+   "pick_run": "false",
+   "same": true,
+   "max_abs_dp": 0.0001364775619506764
+  },
+  {
+   "set": "jevals-pubmedqa",
+   "id": "jevals-pubmedqa:pubmedqa-871",
+   "qid": "decision",
+   "type": "noul",
+   "pick": "false",
+   "pick_run": "false",
+   "same": true,
+   "max_abs_dp": 0.002156544670104954
+  },
+  {
+   "set": "jevals-banking77",
+   "id": "jevals-banking77:banking77-1749",
+   "qid": "decision",
+   "type": "choice",
+   "pick": "failed_transfer",
+   "pick_run": "failed_transfer",
+   "same": true,
+   "max_abs_dp": 0.023940279014587396
+  },
+  {
+   "set": "jevals-banking77",
+   "id": "jevals-banking77:banking77-669",
+   "qid": "decision",
+   "type": "choice",
+   "pick": "pending_top_up",
+   "pick_run": "pending_top_up",
+   "same": true,
+   "max_abs_dp": 0.007160110675811793
+  },
+  {
+   "set": "jevals-banking77",
+   "id": "jevals-banking77:banking77-2687",
+   "qid": "decision",
+   "type": "choice",
+   "pick": "transfer_timing",
+   "pick_run": "transfer_timing",
+   "same": true,
+   "max_abs_dp": 0.0005568832015990788
+  },
+  {
+   "set": "jevals-banking77",
+   "id": "jevals-banking77:banking77-610",
+   "qid": "decision",
+   "type": "choice",
+   "pick": "receiving_money",
+   "pick_run": "receiving_money",
+   "same": true,
+   "max_abs_dp": 0.005873244781494091
+  },
+  {
+   "set": "jevals-banking77",
+   "id": "jevals-banking77:banking77-2279",
+   "qid": "decision",
+   "type": "choice",
+   "pick": "failed_transfer",
+   "pick_run": "failed_transfer",
+   "same": true,
+   "max_abs_dp": 0.003507100162506105
+  },
+  {
+   "set": "jevals-banking77",
+   "id": "jevals-banking77:banking77-2981",
+   "qid": "decision",
+   "type": "choice",
+   "pick": "topping_up_by_card",
+   "pick_run": "topping_up_by_card",
+   "same": true,
+   "max_abs_dp": 0.004674171613693212
+  },
+  {
+   "set": "jevals-banking77",
+   "id": "jevals-banking77:banking77-1242",
+   "qid": "decision",
+   "type": "choice",
+   "pick": "passcode_forgotten",
+   "pick_run": "passcode_forgotten",
+   "same": true,
+   "max_abs_dp": 0.0011834933509826673
+  },
+  {
+   "set": "jevals-banking77",
+   "id": "jevals-banking77:banking77-2986",
+   "qid": "decision",
+   "type": "choice",
+   "pick": "apple_pay_or_google_pay",
+   "pick_run": "apple_pay_or_google_pay",
+   "same": true,
+   "max_abs_dp": 0.011354429265022281
+  },
+  {
+   "set": "jevals-banking77",
+   "id": "jevals-banking77:banking77-2557",
+   "qid": "decision",
+   "type": "choice",
+   "pick": "wrong_exchange_rate_for_cash_withdrawal",
+   "pick_run": "wrong_exchange_rate_for_cash_withdrawal",
+   "same": true,
+   "max_abs_dp": 0.0010959678363799963
+  },
+  {
+   "set": "jevals-banking77",
+   "id": "jevals-banking77:banking77-2076",
+   "qid": "decision",
+   "type": "choice",
+   "pick": "card_not_working",
+   "pick_run": "card_not_working",
+   "same": true,
+   "max_abs_dp": 0.00905289724349978
+  },
+  {
+   "set": "jevals-banking77",
+   "id": "jevals-banking77:banking77-437",
+   "qid": "decision",
+   "type": "choice",
+   "pick": "fiat_currency_support",
+   "pick_run": "fiat_currency_support",
+   "same": true,
+   "max_abs_dp": 0.0004598371599912644
+  },
+  {
+   "set": "jevals-banking77",
+   "id": "jevals-banking77:banking77-2717",
+   "qid": "decision",
+   "type": "choice",
+   "pick": "cash_withdrawal_not_recognised",
+   "pick_run": "cash_withdrawal_not_recognised",
+   "same": true,
+   "max_abs_dp": 0.0032614902958869874
+  },
+  {
+   "set": "jevals-banking77",
+   "id": "jevals-banking77:banking77-1407",
+   "qid": "decision",
+   "type": "choice",
+   "pick": "compromised_card",
+   "pick_run": "compromised_card",
+   "same": true,
+   "max_abs_dp": 0.0003835960665047166
+  },
+  {
+   "set": "jevals-banking77",
+   "id": "jevals-banking77:banking77-51",
+   "qid": "decision",
+   "type": "choice",
+   "pick": "card_linking",
+   "pick_run": "card_linking",
+   "same": true,
+   "max_abs_dp": 0.0003765650672912235
+  },
+  {
+   "set": "jevals-banking77",
+   "id": "jevals-banking77:banking77-1566",
+   "qid": "decision",
+   "type": "choice",
+   "pick": "declined_cash_withdrawal",
+   "pick_run": "declined_cash_withdrawal",
+   "same": true,
+   "max_abs_dp": 0.007276423375129709
+  },
+  {
+   "set": "jevals-banking77",
+   "id": "jevals-banking77:banking77-1817",
+   "qid": "decision",
+   "type": "choice",
+   "pick": "declined_card_payment",
+   "pick_run": "declined_card_payment",
+   "same": true,
+   "max_abs_dp": 0.008562368899822231
+  },
+  {
+   "set": "jevals-banking77",
+   "id": "jevals-banking77:banking77-1244",
+   "qid": "decision",
+   "type": "choice",
+   "pick": "passcode_forgotten",
+   "pick_run": "passcode_forgotten",
+   "same": true,
+   "max_abs_dp": 0.021577001438140897
+  },
+  {
+   "set": "jevals-banking77",
+   "id": "jevals-banking77:banking77-1169",
+   "qid": "decision",
+   "type": "choice",
+   "pick": "unable_to_verify_identity",
+   "pick_run": "unable_to_verify_identity",
+   "same": true,
+   "max_abs_dp": 0.0033154890518188873
+  },
+  {
+   "set": "jevals-banking77",
+   "id": "jevals-banking77:banking77-2430",
+   "qid": "decision",
+   "type": "choice",
+   "pick": "top_up_by_card_charge",
+   "pick_run": "top_up_by_card_charge",
+   "same": true,
+   "max_abs_dp": 0.015199531869888316
+  },
+  {
+   "set": "jevals-banking77",
+   "id": "jevals-banking77:banking77-3073",
+   "qid": "decision",
+   "type": "choice",
+   "pick": "country_support",
+   "pick_run": "country_support",
+   "same": true,
+   "max_abs_dp": 0.00039297395324711104
+  },
+  {
+   "set": "jevals-banking77",
+   "id": "jevals-banking77:banking77-10",
+   "qid": "decision",
+   "type": "choice",
+   "pick": "lost_or_stolen_card",
+   "pick_run": "lost_or_stolen_card",
+   "same": true,
+   "max_abs_dp": 0.018477167479515083
+  },
+  {
+   "set": "jevals-banking77",
+   "id": "jevals-banking77:banking77-578",
+   "qid": "decision",
+   "type": "choice",
+   "pick": "contactless_not_working",
+   "pick_run": "contactless_not_working",
+   "same": true,
+   "max_abs_dp": 0.001786226730346674
+  },
+  {
+   "set": "jevals-banking77",
+   "id": "jevals-banking77:banking77-1000",
+   "qid": "decision",
+   "type": "choice",
+   "pick": "top_up_reverted",
+   "pick_run": "top_up_reverted",
+   "same": true,
+   "max_abs_dp": 0.004602908008575435
+  },
+  {
+   "set": "jevals-banking77",
+   "id": "jevals-banking77:banking77-2650",
+   "qid": "decision",
+   "type": "choice",
+   "pick": "top_up_failed",
+   "pick_run": "top_up_failed",
+   "same": true,
+   "max_abs_dp": 0.0017825746154784827
+  },
+  {
+   "set": "jevals-banking77",
+   "id": "jevals-banking77:banking77-1341",
+   "qid": "decision",
+   "type": "choice",
+   "pick": "topping_up_by_card",
+   "pick_run": "topping_up_by_card",
+   "same": true,
+   "max_abs_dp": 0.0066273999137878015
+  },
+  {
+   "set": "jevals-banking77",
+   "id": "jevals-banking77:banking77-3019",
+   "qid": "decision",
+   "type": "choice",
+   "pick": "verify_my_identity",
+   "pick_run": "verify_my_identity",
+   "same": true,
+   "max_abs_dp": 0.01950413126373296
+  },
+  {
+   "set": "jevals-banking77",
+   "id": "jevals-banking77:banking77-2199",
+   "qid": "decision",
+   "type": "choice",
+   "pick": "transfer_fee_charged",
+   "pick_run": "transfer_fee_charged",
+   "same": true,
+   "max_abs_dp": 0.0006088438873290691
+  },
+  {
+   "set": "jevals-banking77",
+   "id": "jevals-banking77:banking77-1719",
+   "qid": "decision",
+   "type": "choice",
+   "pick": "failed_transfer",
+   "pick_run": "failed_transfer",
+   "same": true,
+   "max_abs_dp": 0.003761896965026823
+  },
+  {
+   "set": "jevals-banking77",
+   "id": "jevals-banking77:banking77-1055",
+   "qid": "decision",
+   "type": "choice",
+   "pick": "balance_not_updated_after_cheque_or_cash_deposit",
+   "pick_run": "balance_not_updated_after_cheque_or_cash_deposit",
+   "same": true,
+   "max_abs_dp": 0.0021776530952453133
+  },
+  {
+   "set": "jevals-banking77",
+   "id": "jevals-banking77:banking77-1227",
+   "qid": "decision",
+   "type": "choice",
+   "pick": "unable_to_verify_identity",
+   "pick_run": "unable_to_verify_identity",
+   "same": true,
+   "max_abs_dp": 0.02141378558158874
+  },
+  {
+   "set": "jevals-banking77",
+   "id": "jevals-banking77:banking77-2237",
+   "qid": "decision",
+   "type": "choice",
+   "pick": "fiat_currency_support",
+   "pick_run": "fiat_currency_support",
+   "same": true,
+   "max_abs_dp": 0.0026266516380309968
+  },
+  {
+   "set": "jevals-banking77",
+   "id": "jevals-banking77:banking77-2633",
+   "qid": "decision",
+   "type": "choice",
+   "pick": "disposable_card_limits",
+   "pick_run": "disposable_card_limits",
+   "same": true,
+   "max_abs_dp": 0.00017110443878176618
+  },
+  {
+   "set": "jevals-banking77",
+   "id": "jevals-banking77:banking77-2362",
+   "qid": "decision",
+   "type": "choice",
+   "pick": "verify_top_up",
+   "pick_run": "verify_top_up",
+   "same": true,
+   "max_abs_dp": 0.0018698823356628758
+  },
+  {
+   "set": "jevals-banking77",
+   "id": "jevals-banking77:banking77-157",
+   "qid": "decision",
+   "type": "choice",
+   "pick": "card_payment_wrong_exchange_rate",
+   "pick_run": "card_payment_wrong_exchange_rate",
+   "same": true,
+   "max_abs_dp": 0.019179564340591426
+  },
+  {
+   "set": "jevals-banking77",
+   "id": "jevals-banking77:banking77-396",
+   "qid": "decision",
+   "type": "choice",
+   "pick": "card_acceptance",
+   "pick_run": "card_acceptance",
+   "same": true,
+   "max_abs_dp": 0.017113198209762592
+  },
+  {
+   "set": "jevals-banking77",
+   "id": "jevals-banking77:banking77-472",
+   "qid": "decision",
+   "type": "choice",
+   "pick": "lost_or_stolen_card",
+   "pick_run": "lost_or_stolen_card",
+   "same": true,
+   "max_abs_dp": 0.005541816398620614
+  },
+  {
+   "set": "jevals-banking77",
+   "id": "jevals-banking77:banking77-1528",
+   "qid": "decision",
+   "type": "choice",
+   "pick": "passcode_forgotten",
+   "pick_run": "passcode_forgotten",
+   "same": true,
+   "max_abs_dp": 8.043476058542721e-05
+  },
+  {
+   "set": "jevals-banking77",
+   "id": "jevals-banking77:banking77-2432",
+   "qid": "decision",
+   "type": "choice",
+   "pick": "getting_spare_card",
+   "pick_run": "getting_spare_card",
+   "same": true,
+   "max_abs_dp": 0.0015329588203430444
+  },
+  {
+   "set": "jevals-banking77",
+   "id": "jevals-banking77:banking77-819",
+   "qid": "decision",
+   "type": "choice",
+   "pick": "extra_charge_on_statement",
+   "pick_run": "extra_charge_on_statement",
+   "same": true,
+   "max_abs_dp": 0.0037726735286712776
+  },
+  {
+   "set": "jevals-banking77",
+   "id": "jevals-banking77:banking77-1504",
+   "qid": "decision",
+   "type": "choice",
+   "pick": "direct_debit_payment_not_recognised",
+   "pick_run": "direct_debit_payment_not_recognised",
+   "same": true,
+   "max_abs_dp": 0.00045620281219482806
+  },
+  {
+   "set": "jevals-banking77",
+   "id": "jevals-banking77:banking77-978",
+   "qid": "decision",
+   "type": "choice",
+   "pick": "card_acceptance",
+   "pick_run": "card_acceptance",
+   "same": true,
+   "max_abs_dp": 0.005841841838598254
+  },
+  {
+   "set": "jevals-banking77",
+   "id": "jevals-banking77:banking77-2236",
+   "qid": "decision",
+   "type": "choice",
+   "pick": "receiving_money",
+   "pick_run": "receiving_money",
+   "same": true,
+   "max_abs_dp": 0.0032306124305725437
+  },
+  {
+   "set": "jevals-banking77",
+   "id": "jevals-banking77:banking77-1925",
+   "qid": "decision",
+   "type": "choice",
+   "pick": "card_swallowed",
+   "pick_run": "card_swallowed",
+   "same": true,
+   "max_abs_dp": 0.00024314561891555786
+  },
+  {
+   "set": "jevals-banking77",
+   "id": "jevals-banking77:banking77-704",
+   "qid": "decision",
+   "type": "choice",
+   "pick": "cancel_transfer",
+   "pick_run": "cancel_transfer",
+   "same": true,
+   "max_abs_dp": 0.01998433731365204
+  },
+  {
+   "set": "jevals-banking77",
+   "id": "jevals-banking77:banking77-2288",
+   "qid": "decision",
+   "type": "choice",
+   "pick": "failed_transfer",
+   "pick_run": "failed_transfer",
+   "same": true,
+   "max_abs_dp": 0.0016355493559837356
+  },
+  {
+   "set": "jevals-banking77",
+   "id": "jevals-banking77:banking77-2119",
+   "qid": "decision",
+   "type": "choice",
+   "pick": "change_pin",
+   "pick_run": "change_pin",
+   "same": true,
+   "max_abs_dp": 0.0011953313136100763
+  },
+  {
+   "set": "jevals-banking77",
+   "id": "jevals-banking77:banking77-2886",
+   "qid": "decision",
+   "type": "choice",
+   "pick": "extra_charge_on_statement",
+   "pick_run": "extra_charge_on_statement",
+   "same": true,
+   "max_abs_dp": 0.010501723793029827
+  },
+  {
+   "set": "jevals-banking77",
+   "id": "jevals-banking77:banking77-1931",
+   "qid": "decision",
+   "type": "choice",
+   "pick": "card_swallowed",
+   "pick_run": "card_swallowed",
+   "same": true,
+   "max_abs_dp": 0.0027530267868042246
+  },
+  {
+   "set": "jevals-banking77",
+   "id": "jevals-banking77:banking77-2558",
+   "qid": "decision",
+   "type": "choice",
+   "pick": "cash_withdrawal_charge",
+   "pick_run": "cash_withdrawal_charge",
+   "same": true,
+   "max_abs_dp": 0.005529832390785215
+  },
+  {
+   "set": "jevals-banking77",
+   "id": "jevals-banking77:banking77-2801",
+   "qid": "decision",
+   "type": "choice",
+   "pick": "top_up_by_card_charge",
+   "pick_run": "top_up_by_card_charge",
+   "same": true,
+   "max_abs_dp": 0.006675042510032647
+  },
+  {
+   "set": "jevals-banking77",
+   "id": "jevals-banking77:banking77-2842",
+   "qid": "decision",
+   "type": "choice",
+   "pick": "activate_my_card",
+   "pick_run": "activate_my_card",
+   "same": true,
+   "max_abs_dp": 0.00021903362655639746
+  },
+  {
+   "set": "jevals-banking77",
+   "id": "jevals-banking77:banking77-258",
+   "qid": "decision",
+   "type": "choice",
+   "pick": "fiat_currency_support",
+   "pick_run": "fiat_currency_support",
+   "same": true,
+   "max_abs_dp": 0.005284831920623789
+  },
+  {
+   "set": "jevals-banking77",
+   "id": "jevals-banking77:banking77-180",
+   "qid": "decision",
+   "type": "choice",
+   "pick": "extra_charge_on_statement",
+   "pick_run": "extra_charge_on_statement",
+   "same": true,
+   "max_abs_dp": 0.005434870723724394
+  },
+  {
+   "set": "jevals-banking77",
+   "id": "jevals-banking77:banking77-2691",
+   "qid": "decision",
+   "type": "choice",
+   "pick": "balance_not_updated_after_bank_transfer",
+   "pick_run": "balance_not_updated_after_bank_transfer",
+   "same": true,
+   "max_abs_dp": 0.0010594485855102143
+  },
+  {
+   "set": "jevals-banking77",
+   "id": "jevals-banking77:banking77-634",
+   "qid": "decision",
+   "type": "choice",
+   "pick": "top_up_by_bank_transfer_charge",
+   "pick_run": "top_up_by_bank_transfer_charge",
+   "same": true,
+   "max_abs_dp": 0.0021456788978576613
+  },
+  {
+   "set": "jevals-banking77",
+   "id": "jevals-banking77:banking77-2958",
+   "qid": "decision",
+   "type": "choice",
+   "pick": "top_up_failed",
+   "pick_run": "top_up_failed",
+   "same": true,
+   "max_abs_dp": 0.0007090187530517378
+  },
+  {
+   "set": "jevals-banking77",
+   "id": "jevals-banking77:banking77-882",
+   "qid": "decision",
+   "type": "choice",
+   "pick": "supported_cards_and_currencies",
+   "pick_run": "fiat_currency_support",
+   "same": false,
+   "max_abs_dp": 0.01041107138633729
+  },
+  {
+   "set": "jevals-banking77",
+   "id": "jevals-banking77:banking77-218",
+   "qid": "decision",
+   "type": "choice",
+   "pick": "pending_cash_withdrawal",
+   "pick_run": "pending_cash_withdrawal",
+   "same": true,
+   "max_abs_dp": 0.004912816371917761
+  },
+  {
+   "set": "jevals-banking77",
+   "id": "jevals-banking77:banking77-1871",
+   "qid": "decision",
+   "type": "choice",
+   "pick": "transfer_fee_charged",
+   "pick_run": "transfer_fee_charged",
+   "same": true,
+   "max_abs_dp": 0.020876409551620456
+  },
+  {
+   "set": "jevals-banking77",
+   "id": "jevals-banking77:banking77-1660",
+   "qid": "decision",
+   "type": "choice",
+   "pick": "lost_or_stolen_card",
+   "pick_run": "lost_or_stolen_card",
+   "same": true,
+   "max_abs_dp": 0.00979334916305541
+  },
+  {
+   "set": "jevals-banking77",
+   "id": "jevals-banking77:banking77-1126",
+   "qid": "decision",
+   "type": "choice",
+   "pick": "edit_personal_details",
+   "pick_run": "edit_personal_details",
+   "same": true,
+   "max_abs_dp": 0.00035575588607783537
+  },
+  {
+   "set": "jevals-banking77",
+   "id": "jevals-banking77:banking77-2321",
+   "qid": "decision",
+   "type": "choice",
+   "pick": "transfer_into_account",
+   "pick_run": "transfer_into_account",
+   "same": true,
+   "max_abs_dp": 0.00010597671341151002
+  },
+  {
+   "set": "jevals-banking77",
+   "id": "jevals-banking77:banking77-2667",
+   "qid": "decision",
+   "type": "choice",
+   "pick": "top_up_failed",
+   "pick_run": "top_up_failed",
+   "same": true,
+   "max_abs_dp": 0.0010350030363202103
+  },
+  {
+   "set": "jevals-banking77",
+   "id": "jevals-banking77:banking77-1537",
+   "qid": "decision",
+   "type": "choice",
+   "pick": "passcode_forgotten",
+   "pick_run": "passcode_forgotten",
+   "same": true,
+   "max_abs_dp": 0.012686089736938522
+  },
+  {
+   "set": "jevals-banking77",
+   "id": "jevals-banking77:banking77-1232",
+   "qid": "decision",
+   "type": "choice",
+   "pick": "why_verify_identity",
+   "pick_run": "why_verify_identity",
+   "same": true,
+   "max_abs_dp": 0.01133909769821162
+  },
+  {
+   "set": "jevals-banking77",
+   "id": "jevals-banking77:banking77-2490",
+   "qid": "decision",
+   "type": "choice",
+   "pick": "get_physical_card",
+   "pick_run": "get_physical_card",
+   "same": true,
+   "max_abs_dp": 0.0013011589164734216
+  },
+  {
+   "set": "jevals-banking77",
+   "id": "jevals-banking77:banking77-1156",
+   "qid": "decision",
+   "type": "choice",
+   "pick": "edit_personal_details",
+   "pick_run": "edit_personal_details",
+   "same": true,
+   "max_abs_dp": 0.0003208802604675043
+  },
+  {
+   "set": "jevals-banking77",
+   "id": "jevals-banking77:banking77-2020",
+   "qid": "decision",
+   "type": "choice",
+   "pick": "verify_source_of_funds",
+   "pick_run": "verify_source_of_funds",
+   "same": true,
+   "max_abs_dp": 0.009149723342895544
+  },
+  {
+   "set": "jevals-banking77",
+   "id": "jevals-banking77:banking77-1671",
+   "qid": "decision",
+   "type": "choice",
+   "pick": "lost_or_stolen_phone",
+   "pick_run": "lost_or_stolen_phone",
+   "same": true,
+   "max_abs_dp": 0.009801792629241968
+  },
+  {
+   "set": "jevals-banking77",
+   "id": "jevals-banking77:banking77-2933",
+   "qid": "decision",
+   "type": "choice",
+   "pick": "order_physical_card",
+   "pick_run": "order_physical_card",
+   "same": true,
+   "max_abs_dp": 0.0070786974754333265
+  },
+  {
+   "set": "jevals-helpsteer2",
+   "id": "jevals-helpsteer2:helpsteer2-599",
+   "qid": "decision",
+   "type": "score",
+   "pick": "3",
+   "pick_run": "3",
+   "same": true,
+   "max_abs_dp": 0.0017070311212539768
+  },
+  {
+   "set": "jevals-helpsteer2",
+   "id": "jevals-helpsteer2:helpsteer2-269",
+   "qid": "decision",
+   "type": "score",
+   "pick": "4",
+   "pick_run": "4",
+   "same": true,
+   "max_abs_dp": 0.004396505262374895
+  },
+  {
+   "set": "jevals-helpsteer2",
+   "id": "jevals-helpsteer2:helpsteer2-910",
+   "qid": "decision",
+   "type": "score",
+   "pick": "2",
+   "pick_run": "2",
+   "same": true,
+   "max_abs_dp": 0.001993250860214235
+  },
+  {
+   "set": "jevals-helpsteer2",
+   "id": "jevals-helpsteer2:helpsteer2-243",
+   "qid": "decision",
+   "type": "score",
+   "pick": "4",
+   "pick_run": "4",
+   "same": true,
+   "max_abs_dp": 0.0022569894447326266
+  },
+  {
+   "set": "jevals-helpsteer2",
+   "id": "jevals-helpsteer2:helpsteer2-789",
+   "qid": "decision",
+   "type": "score",
+   "pick": "1",
+   "pick_run": "1",
+   "same": true,
+   "max_abs_dp": 0.0014194283542633135
+  },
+  {
+   "set": "jevals-helpsteer2",
+   "id": "jevals-helpsteer2:helpsteer2-1001",
+   "qid": "decision",
+   "type": "score",
+   "pick": "0",
+   "pick_run": "0",
+   "same": true,
+   "max_abs_dp": 0.0025998412761688017
+  },
+  {
+   "set": "jevals-helpsteer2",
+   "id": "jevals-helpsteer2:helpsteer2-448",
+   "qid": "decision",
+   "type": "score",
+   "pick": "3",
+   "pick_run": "3",
+   "same": true,
+   "max_abs_dp": 0.0021863538465499988
+  },
+  {
+   "set": "jevals-helpsteer2",
+   "id": "jevals-helpsteer2:helpsteer2-1008",
+   "qid": "decision",
+   "type": "score",
+   "pick": "4",
+   "pick_run": "4",
+   "same": true,
+   "max_abs_dp": 0.0024829814634323155
+  },
+  {
+   "set": "jevals-helpsteer2",
+   "id": "jevals-helpsteer2:helpsteer2-873",
+   "qid": "decision",
+   "type": "score",
+   "pick": "4",
+   "pick_run": "4",
+   "same": true,
+   "max_abs_dp": 0.001164060227394098
+  },
+  {
+   "set": "jevals-helpsteer2",
+   "id": "jevals-helpsteer2:helpsteer2-705",
+   "qid": "decision",
+   "type": "score",
+   "pick": "1",
+   "pick_run": "1",
+   "same": true,
+   "max_abs_dp": 0.002851796442985527
+  },
+  {
+   "set": "jevals-helpsteer2",
+   "id": "jevals-helpsteer2:helpsteer2-151",
+   "qid": "decision",
+   "type": "score",
+   "pick": "2",
+   "pick_run": "2",
+   "same": true,
+   "max_abs_dp": 0.0020692041740417444
+  },
+  {
+   "set": "jevals-helpsteer2",
+   "id": "jevals-helpsteer2:helpsteer2-920",
+   "qid": "decision",
+   "type": "score",
+   "pick": "0",
+   "pick_run": "0",
+   "same": true,
+   "max_abs_dp": 0.004536616085052514
+  },
+  {
+   "set": "jevals-helpsteer2",
+   "id": "jevals-helpsteer2:helpsteer2-496",
+   "qid": "decision",
+   "type": "score",
+   "pick": "4",
+   "pick_run": "4",
+   "same": true,
+   "max_abs_dp": 0.0011169719676971668
+  },
+  {
+   "set": "jevals-helpsteer2",
+   "id": "jevals-helpsteer2:helpsteer2-21",
+   "qid": "decision",
+   "type": "score",
+   "pick": "0",
+   "pick_run": "0",
+   "same": true,
+   "max_abs_dp": 0.0010305535545349631
+  },
+  {
+   "set": "jevals-helpsteer2",
+   "id": "jevals-helpsteer2:helpsteer2-553",
+   "qid": "decision",
+   "type": "score",
+   "pick": "4",
+   "pick_run": "4",
+   "same": true,
+   "max_abs_dp": 0.0019170843048095798
+  },
+  {
+   "set": "jevals-helpsteer2",
+   "id": "jevals-helpsteer2:helpsteer2-618",
+   "qid": "decision",
+   "type": "score",
+   "pick": "4",
+   "pick_run": "4",
+   "same": true,
+   "max_abs_dp": 0.0021986666736602545
+  },
+  {
+   "set": "jevals-helpsteer2",
+   "id": "jevals-helpsteer2:helpsteer2-452",
+   "qid": "decision",
+   "type": "score",
+   "pick": "0",
+   "pick_run": "0",
+   "same": true,
+   "max_abs_dp": 0.0028643911151886026
+  },
+  {
+   "set": "jevals-helpsteer2",
+   "id": "jevals-helpsteer2:helpsteer2-422",
+   "qid": "decision",
+   "type": "score",
+   "pick": "3",
+   "pick_run": "3",
+   "same": true,
+   "max_abs_dp": 0.0068150336418151825
+  },
+  {
+   "set": "jevals-helpsteer2",
+   "id": "jevals-helpsteer2:helpsteer2-820",
+   "qid": "decision",
+   "type": "score",
+   "pick": "4",
+   "pick_run": "4",
+   "same": true,
+   "max_abs_dp": 0.003520079170227064
+  },
+  {
+   "set": "jevals-helpsteer2",
+   "id": "jevals-helpsteer2:helpsteer2-1034",
+   "qid": "decision",
+   "type": "score",
+   "pick": "3",
+   "pick_run": "3",
+   "same": true,
+   "max_abs_dp": 0.003428257497787468
+  },
+  {
+   "set": "jevals-helpsteer2",
+   "id": "jevals-helpsteer2:helpsteer2-14",
+   "qid": "decision",
+   "type": "score",
+   "pick": "4",
+   "pick_run": "4",
+   "same": true,
+   "max_abs_dp": 0.0030382548294067835
+  },
+  {
+   "set": "jevals-helpsteer2",
+   "id": "jevals-helpsteer2:helpsteer2-221",
+   "qid": "decision",
+   "type": "score",
+   "pick": "3",
+   "pick_run": "3",
+   "same": true,
+   "max_abs_dp": 0.002335368223190304
+  },
+  {
+   "set": "jevals-helpsteer2",
+   "id": "jevals-helpsteer2:helpsteer2-367",
+   "qid": "decision",
+   "type": "score",
+   "pick": "3",
+   "pick_run": "3",
+   "same": true,
+   "max_abs_dp": 0.005168520936965931
+  },
+  {
+   "set": "jevals-helpsteer2",
+   "id": "jevals-helpsteer2:helpsteer2-898",
+   "qid": "decision",
+   "type": "score",
+   "pick": "1",
+   "pick_run": "1",
+   "same": true,
+   "max_abs_dp": 0.003371750524520878
+  },
+  {
+   "set": "jevals-helpsteer2",
+   "id": "jevals-helpsteer2:helpsteer2-475",
+   "qid": "decision",
+   "type": "score",
+   "pick": "3",
+   "pick_run": "3",
+   "same": true,
+   "max_abs_dp": 0.0018364327745437614
+  },
+  {
+   "set": "jevals-helpsteer2",
+   "id": "jevals-helpsteer2:helpsteer2-1022",
+   "qid": "decision",
+   "type": "score",
+   "pick": "3",
+   "pick_run": "3",
+   "same": true,
+   "max_abs_dp": 0.003297201475143452
+  },
+  {
+   "set": "jevals-helpsteer2",
+   "id": "jevals-helpsteer2:helpsteer2-767",
+   "qid": "decision",
+   "type": "score",
+   "pick": "2",
+   "pick_run": "2",
+   "same": true,
+   "max_abs_dp": 0.00298056771183014
+  },
+  {
+   "set": "jevals-helpsteer2",
+   "id": "jevals-helpsteer2:helpsteer2-588",
+   "qid": "decision",
+   "type": "score",
+   "pick": "4",
+   "pick_run": "4",
+   "same": true,
+   "max_abs_dp": 0.0030666067352295157
+  },
+  {
+   "set": "jevals-helpsteer2",
+   "id": "jevals-helpsteer2:helpsteer2-380",
+   "qid": "decision",
+   "type": "score",
+   "pick": "4",
+   "pick_run": "4",
+   "same": true,
+   "max_abs_dp": 0.0031253642139434934
+  },
+  {
+   "set": "jevals-helpsteer2",
+   "id": "jevals-helpsteer2:helpsteer2-444",
+   "qid": "decision",
+   "type": "score",
+   "pick": "2",
+   "pick_run": "2",
+   "same": true,
+   "max_abs_dp": 0.0029246465024947987
+  },
+  {
+   "set": "jevals-helpsteer2",
+   "id": "jevals-helpsteer2:helpsteer2-786",
+   "qid": "decision",
+   "type": "score",
+   "pick": "1",
+   "pick_run": "1",
+   "same": true,
+   "max_abs_dp": 0.002068277301788324
+  },
+  {
+   "set": "jevals-helpsteer2",
+   "id": "jevals-helpsteer2:helpsteer2-891",
+   "qid": "decision",
+   "type": "score",
+   "pick": "0",
+   "pick_run": "0",
+   "same": true,
+   "max_abs_dp": 0.0015666385917663117
+  },
+  {
+   "set": "jevals-helpsteer2",
+   "id": "jevals-helpsteer2:helpsteer2-805",
+   "qid": "decision",
+   "type": "score",
+   "pick": "1",
+   "pick_run": "1",
+   "same": true,
+   "max_abs_dp": 0.0020603434896468986
+  },
+  {
+   "set": "jevals-helpsteer2",
+   "id": "jevals-helpsteer2:helpsteer2-54",
+   "qid": "decision",
+   "type": "score",
+   "pick": "4",
+   "pick_run": "4",
+   "same": true,
+   "max_abs_dp": 0.005909478620529179
+  },
+  {
+   "set": "jevals-helpsteer2",
+   "id": "jevals-helpsteer2:helpsteer2-142",
+   "qid": "decision",
+   "type": "score",
+   "pick": "3",
+   "pick_run": "3",
+   "same": true,
+   "max_abs_dp": 0.004755552507400507
+  },
+  {
+   "set": "jevals-helpsteer2",
+   "id": "jevals-helpsteer2:helpsteer2-178",
+   "qid": "decision",
+   "type": "score",
+   "pick": "1",
+   "pick_run": "1",
+   "same": true,
+   "max_abs_dp": 0.0042735126762389886
+  },
+  {
+   "set": "jevals-helpsteer2",
+   "id": "jevals-helpsteer2:helpsteer2-531",
+   "qid": "decision",
+   "type": "score",
+   "pick": "4",
+   "pick_run": "4",
+   "same": true,
+   "max_abs_dp": 0.001332084762573238
+  },
+  {
+   "set": "jevals-helpsteer2",
+   "id": "jevals-helpsteer2:helpsteer2-823",
+   "qid": "decision",
+   "type": "score",
+   "pick": "4",
+   "pick_run": "4",
+   "same": true,
+   "max_abs_dp": 0.004615197769165025
+  },
+  {
+   "set": "jevals-helpsteer2",
+   "id": "jevals-helpsteer2:helpsteer2-305",
+   "qid": "decision",
+   "type": "score",
+   "pick": "4",
+   "pick_run": "4",
+   "same": true,
+   "max_abs_dp": 0.0016935211257934624
+  },
+  {
+   "set": "jevals-helpsteer2",
+   "id": "jevals-helpsteer2:helpsteer2-526",
+   "qid": "decision",
+   "type": "score",
+   "pick": "4",
+   "pick_run": "4",
+   "same": true,
+   "max_abs_dp": 0.002539206399917615
+  },
+  {
+   "set": "jevals-helpsteer2",
+   "id": "jevals-helpsteer2:helpsteer2-359",
+   "qid": "decision",
+   "type": "score",
+   "pick": "4",
+   "pick_run": "4",
+   "same": true,
+   "max_abs_dp": 0.0010118418216705227
+  },
+  {
+   "set": "jevals-helpsteer2",
+   "id": "jevals-helpsteer2:helpsteer2-783",
+   "qid": "decision",
+   "type": "score",
+   "pick": "1",
+   "pick_run": "1",
+   "same": true,
+   "max_abs_dp": 0.0010982502269744898
+  },
+  {
+   "set": "jevals-helpsteer2",
+   "id": "jevals-helpsteer2:helpsteer2-644",
+   "qid": "decision",
+   "type": "score",
+   "pick": "0",
+   "pick_run": "0",
+   "same": true,
+   "max_abs_dp": 0.0013904987869262642
+  },
+  {
+   "set": "jevals-helpsteer2",
+   "id": "jevals-helpsteer2:helpsteer2-278",
+   "qid": "decision",
+   "type": "score",
+   "pick": "1",
+   "pick_run": "1",
+   "same": true,
+   "max_abs_dp": 0.0006149902935028151
+  },
+  {
+   "set": "jevals-helpsteer2",
+   "id": "jevals-helpsteer2:helpsteer2-790",
+   "qid": "decision",
+   "type": "score",
+   "pick": "4",
+   "pick_run": "4",
+   "same": true,
+   "max_abs_dp": 0.0033819403591155983
+  },
+  {
+   "set": "jevals-helpsteer2",
+   "id": "jevals-helpsteer2:helpsteer2-723",
+   "qid": "decision",
+   "type": "score",
+   "pick": "4",
+   "pick_run": "4",
+   "same": true,
+   "max_abs_dp": 0.000605950281143186
+  },
+  {
+   "set": "jevals-helpsteer2",
+   "id": "jevals-helpsteer2:helpsteer2-972",
+   "qid": "decision",
+   "type": "score",
+   "pick": "3",
+   "pick_run": "2",
+   "same": false,
+   "max_abs_dp": 0.007170431488037116
+  },
+  {
+   "set": "jevals-helpsteer2",
+   "id": "jevals-helpsteer2:helpsteer2-646",
+   "qid": "decision",
+   "type": "score",
+   "pick": "3",
+   "pick_run": "3",
+   "same": true,
+   "max_abs_dp": 0.006493632644653302
+  },
+  {
+   "set": "jevals-helpsteer2",
+   "id": "jevals-helpsteer2:helpsteer2-878",
+   "qid": "decision",
+   "type": "score",
+   "pick": "3",
+   "pick_run": "3",
+   "same": true,
+   "max_abs_dp": 0.0025942115430832025
+  },
+  {
+   "set": "jevals-helpsteer2",
+   "id": "jevals-helpsteer2:helpsteer2-945",
+   "qid": "decision",
+   "type": "score",
+   "pick": "2",
+   "pick_run": "3",
+   "same": false,
+   "max_abs_dp": 0.003689604001998914
+  },
+  {
+   "set": "jevals-helpsteer2",
+   "id": "jevals-helpsteer2:helpsteer2-963",
+   "qid": "decision",
+   "type": "score",
+   "pick": "4",
+   "pick_run": "4",
+   "same": true,
+   "max_abs_dp": 0.0027546647338866914
+  },
+  {
+   "set": "jevals-helpsteer2",
+   "id": "jevals-helpsteer2:helpsteer2-83",
+   "qid": "decision",
+   "type": "score",
+   "pick": "3",
+   "pick_run": "3",
+   "same": true,
+   "max_abs_dp": 0.003922615794181822
+  },
+  {
+   "set": "jevals-helpsteer2",
+   "id": "jevals-helpsteer2:helpsteer2-66",
+   "qid": "decision",
+   "type": "score",
+   "pick": "3",
+   "pick_run": "3",
+   "same": true,
+   "max_abs_dp": 0.0036037472667694304
+  },
+  {
+   "set": "jevals-helpsteer2",
+   "id": "jevals-helpsteer2:helpsteer2-914",
+   "qid": "decision",
+   "type": "score",
+   "pick": "3",
+   "pick_run": "3",
+   "same": true,
+   "max_abs_dp": 0.0057809354381561096
+  },
+  {
+   "set": "jevals-helpsteer2",
+   "id": "jevals-helpsteer2:helpsteer2-258",
+   "qid": "decision",
+   "type": "score",
+   "pick": "4",
+   "pick_run": "4",
+   "same": true,
+   "max_abs_dp": 0.0025659412574768337
+  },
+  {
+   "set": "jevals-helpsteer2",
+   "id": "jevals-helpsteer2:helpsteer2-998",
+   "qid": "decision",
+   "type": "score",
+   "pick": "1",
+   "pick_run": "1",
+   "same": true,
+   "max_abs_dp": 0.0053724314994811895
+  },
+  {
+   "set": "jevals-helpsteer2",
+   "id": "jevals-helpsteer2:helpsteer2-326",
+   "qid": "decision",
+   "type": "score",
+   "pick": "4",
+   "pick_run": "4",
+   "same": true,
+   "max_abs_dp": 0.0018118249874115033
+  },
+  {
+   "set": "jevals-helpsteer2",
+   "id": "jevals-helpsteer2:helpsteer2-69",
+   "qid": "decision",
+   "type": "score",
+   "pick": "3",
+   "pick_run": "3",
+   "same": true,
+   "max_abs_dp": 0.005150423919677738
+  },
+  {
+   "set": "jevals-helpsteer2",
+   "id": "jevals-helpsteer2:helpsteer2-630",
+   "qid": "decision",
+   "type": "score",
+   "pick": "4",
+   "pick_run": "4",
+   "same": true,
+   "max_abs_dp": 0.0009986610774994076
+  },
+  {
+   "set": "jevals-helpsteer2",
+   "id": "jevals-helpsteer2:helpsteer2-566",
+   "qid": "decision",
+   "type": "score",
+   "pick": "3",
+   "pick_run": "3",
+   "same": true,
+   "max_abs_dp": 0.0033846534290313812
+  },
+  {
+   "set": "jevals-helpsteer2",
+   "id": "jevals-helpsteer2:helpsteer2-404",
+   "qid": "decision",
+   "type": "score",
+   "pick": "4",
+   "pick_run": "4",
+   "same": true,
+   "max_abs_dp": 0.0016019409198760881
+  },
+  {
+   "set": "jevals-helpsteer2",
+   "id": "jevals-helpsteer2:helpsteer2-793",
+   "qid": "decision",
+   "type": "score",
+   "pick": "4",
+   "pick_run": "4",
+   "same": true,
+   "max_abs_dp": 0.0037356444110870357
+  },
+  {
+   "set": "jevals-helpsteer2",
+   "id": "jevals-helpsteer2:helpsteer2-901",
+   "qid": "decision",
+   "type": "score",
+   "pick": "0",
+   "pick_run": "0",
+   "same": true,
+   "max_abs_dp": 0.0011874838867187432
+  },
+  {
+   "set": "jevals-helpsteer2",
+   "id": "jevals-helpsteer2:helpsteer2-532",
+   "qid": "decision",
+   "type": "score",
+   "pick": "3",
+   "pick_run": "3",
+   "same": true,
+   "max_abs_dp": 0.001475205549240105
+  },
+  {
+   "set": "jevals-helpsteer2",
+   "id": "jevals-helpsteer2:helpsteer2-446",
+   "qid": "decision",
+   "type": "score",
+   "pick": "3",
+   "pick_run": "3",
+   "same": true,
+   "max_abs_dp": 0.0035881805105209275
+  },
+  {
+   "set": "jevals-helpsteer2",
+   "id": "jevals-helpsteer2:helpsteer2-844",
+   "qid": "decision",
+   "type": "score",
+   "pick": "3",
+   "pick_run": "3",
+   "same": true,
+   "max_abs_dp": 0.0017071657466888324
+  },
+  {
+   "set": "jevals-helpsteer2",
+   "id": "jevals-helpsteer2:helpsteer2-411",
+   "qid": "decision",
+   "type": "score",
+   "pick": "4",
+   "pick_run": "4",
+   "same": true,
+   "max_abs_dp": 0.0013958457717895278
+  },
+  {
+   "set": "jevals-helpsteer2",
+   "id": "jevals-helpsteer2:helpsteer2-685",
+   "qid": "decision",
+   "type": "score",
+   "pick": "4",
+   "pick_run": "4",
+   "same": true,
+   "max_abs_dp": 0.0019896724929809317
+  },
+  {
+   "set": "jevals-helpsteer2",
+   "id": "jevals-helpsteer2:helpsteer2-569",
+   "qid": "decision",
+   "type": "score",
+   "pick": "1",
+   "pick_run": "1",
+   "same": true,
+   "max_abs_dp": 0.003196729080200189
+  },
+  {
+   "set": "jevals-helpsteer2",
+   "id": "jevals-helpsteer2:helpsteer2-980",
+   "qid": "decision",
+   "type": "score",
+   "pick": "2",
+   "pick_run": "2",
+   "same": true,
+   "max_abs_dp": 0.003319434814453126
+  },
+  {
+   "set": "jevals-helpsteer2",
+   "id": "jevals-helpsteer2:helpsteer2-338",
+   "qid": "decision",
+   "type": "score",
+   "pick": "4",
+   "pick_run": "4",
+   "same": true,
+   "max_abs_dp": 0.0026797406692504744
+  },
+  {
+   "set": "jevals-helpsteer2",
+   "id": "jevals-helpsteer2:helpsteer2-926",
+   "qid": "decision",
+   "type": "score",
+   "pick": "4",
+   "pick_run": "4",
+   "same": true,
+   "max_abs_dp": 0.0032809732131958147
+  },
+  {
+   "set": "jevals-helpsteer2",
+   "id": "jevals-helpsteer2:helpsteer2-369",
+   "qid": "decision",
+   "type": "score",
+   "pick": "4",
+   "pick_run": "4",
+   "same": true,
+   "max_abs_dp": 0.003506187999725352
+  },
+  {
+   "set": "jevals-helpsteer2",
+   "id": "jevals-helpsteer2:helpsteer2-648",
+   "qid": "decision",
+   "type": "score",
+   "pick": "3",
+   "pick_run": "3",
+   "same": true,
+   "max_abs_dp": 0.001363708106040959
+  },
+  {
+   "set": "jevals-helpsteer2",
+   "id": "jevals-helpsteer2:helpsteer2-87",
+   "qid": "decision",
+   "type": "score",
+   "pick": "1",
+   "pick_run": "1",
+   "same": true,
+   "max_abs_dp": 0.002428788396835302
+  },
+  {
+   "set": "jevals-helpsteer2",
+   "id": "jevals-helpsteer2:helpsteer2-727",
+   "qid": "decision",
+   "type": "score",
+   "pick": "3",
+   "pick_run": "3",
+   "same": true,
+   "max_abs_dp": 0.002152754644393934
+  },
+  {
+   "set": "jevals-helpsteer2",
+   "id": "jevals-helpsteer2:helpsteer2-401",
+   "qid": "decision",
+   "type": "score",
+   "pick": "4",
+   "pick_run": "4",
+   "same": true,
+   "max_abs_dp": 0.004214124761581395
+  },
+  {
+   "set": "jevals-helpsteer2",
+   "id": "jevals-helpsteer2:helpsteer2-265",
+   "qid": "decision",
+   "type": "score",
+   "pick": "0",
+   "pick_run": "0",
+   "same": true,
+   "max_abs_dp": 0.0055157948646545885
+  },
+  {
+   "set": "jevals-helpsteer2",
+   "id": "jevals-helpsteer2:helpsteer2-780",
+   "qid": "decision",
+   "type": "score",
+   "pick": "1",
+   "pick_run": "1",
+   "same": true,
+   "max_abs_dp": 0.001469104606628413
+  },
+  {
+   "set": "jevals-helpsteer2",
+   "id": "jevals-helpsteer2:helpsteer2-899",
+   "qid": "decision",
+   "type": "score",
+   "pick": "4",
+   "pick_run": "4",
+   "same": true,
+   "max_abs_dp": 0.0003629968891143698
+  },
+  {
+   "set": "nimble-eval",
+   "id": "nimble-eval:fast-41-diverse-211-020-base",
+   "qid": "decision",
+   "type": "choice",
+   "pick": "city_no_r3",
+   "pick_run": "city_no_r3",
+   "same": true,
+   "max_abs_dp": 0.0024864998931884985
+  },
+  {
+   "set": "nimble-eval",
+   "id": "nimble-eval:scale-diverse-174-003-base",
+   "qid": "decision",
+   "type": "score",
+   "pick": "3",
+   "pick_run": "3",
+   "same": true,
+   "max_abs_dp": 0.002774182624816879
+  },
+  {
+   "set": "nimble-eval",
+   "id": "nimble-eval:fast-43-diverse-169-065-counterfactual",
+   "qid": "decision",
+   "type": "choice",
+   "pick": "deny_low",
+   "pick_run": "deny_low",
+   "same": true,
+   "max_abs_dp": 0.001843123527526802
+  },
+  {
+   "set": "nimble-eval",
+   "id": "nimble-eval:scale-diverse-173-001-base",
+   "qid": "decision",
+   "type": "score",
+   "pick": "2",
+   "pick_run": "2",
+   "same": true,
+   "max_abs_dp": 0.0012800111370086797
+  },
+  {
+   "set": "nimble-eval",
+   "id": "nimble-eval:fast-43-diverse-079-037-counterfactual",
+   "qid": "decision",
+   "type": "choice",
+   "pick": "Proceed \u2014 Medium; route camera setup to teaching assistant",
+   "pick_run": "Proceed \u2014 Medium; route camera setup to teaching assistant",
+   "same": true,
+   "max_abs_dp": 0.008462921947479218
+  },
+  {
+   "set": "nimble-eval",
+   "id": "nimble-eval:fast-43-diverse-171-008-counterfactual",
+   "qid": "decision",
+   "type": "noul",
+   "pick": "false",
+   "pick_run": "false",
+   "same": true,
+   "max_abs_dp": 0.0006690865592956508
+  },
+  {
+   "set": "nimble-eval",
+   "id": "nimble-eval:fast-43-diverse-174-011-base",
+   "qid": "decision",
+   "type": "score",
+   "pick": "2",
+   "pick_run": "2",
+   "same": true,
+   "max_abs_dp": 0.0037998003120422452
+  },
+  {
+   "set": "nimble-eval",
+   "id": "nimble-eval:fast-41-diverse-009-004-base",
+   "qid": "decision",
+   "type": "noul",
+   "pick": "true",
+   "pick_run": "true",
+   "same": true,
+   "max_abs_dp": 0.00020508449935907613
+  },
+  {
+   "set": "nimble-eval",
+   "id": "nimble-eval:fast-43-diverse-171-010-base",
+   "qid": "decision",
+   "type": "noul",
+   "pick": "true",
+   "pick_run": "true",
+   "same": true,
+   "max_abs_dp": 0.0001036642386466266
+  },
+  {
+   "set": "nimble-eval",
+   "id": "nimble-eval:fast-43-diverse-169-012-counterfactual",
+   "qid": "decision",
+   "type": "choice",
+   "pick": "deny_low",
+   "pick_run": "deny_low",
+   "same": true,
+   "max_abs_dp": 0.001906281406402588
+  },
+  {
+   "set": "nimble-eval",
+   "id": "nimble-eval:fast-41-diverse-275-013-counterfactual",
+   "qid": "decision",
+   "type": "score",
+   "pick": "0",
+   "pick_run": "0",
+   "same": true,
+   "max_abs_dp": 0.0015062385268211353
+  },
+  {
+   "set": "nimble-eval",
+   "id": "nimble-eval:scale-diverse-097-005-base",
+   "qid": "decision",
+   "type": "choice",
+   "pick": "assign_420_as_best_fit",
+   "pick_run": "assign_420_as_best_fit",
+   "same": true,
+   "max_abs_dp": 0.001358479156494119
+  },
+  {
+   "set": "nimble-eval",
+   "id": "nimble-eval:fast-43-diverse-169-068-base",
+   "qid": "decision",
+   "type": "choice",
+   "pick": "release_high",
+   "pick_run": "release_high",
+   "same": true,
+   "max_abs_dp": 0.005447728443145783
+  },
+  {
+   "set": "nimble-eval",
+   "id": "nimble-eval:fast-41-diverse-079-012-counterfactual",
+   "qid": "decision",
+   "type": "choice",
+   "pick": "Proceed \u2014 High; no remaining routing",
+   "pick_run": "Proceed \u2014 High; no remaining routing",
+   "same": true,
+   "max_abs_dp": 0.005806731111526464
+  },
+  {
+   "set": "nimble-eval",
+   "id": "nimble-eval:scale-diverse-007-005-base",
+   "qid": "decision",
+   "type": "choice",
+   "pick": "major_seller_correction_not_ready",
+   "pick_run": "major_seller_correction_not_ready",
+   "same": true,
+   "max_abs_dp": 0.0009021766853332114
+  },
+  {
+   "set": "nimble-eval",
+   "id": "nimble-eval:fast-41-diverse-171-003-counterfactual",
+   "qid": "decision",
+   "type": "noul",
+   "pick": "false",
+   "pick_run": "false",
+   "same": true,
+   "max_abs_dp": 0.00016909534835818363
+  },
+  {
+   "set": "nimble-eval",
+   "id": "nimble-eval:fast-41-diverse-212-032-counterfactual",
+   "qid": "decision",
+   "type": "choice",
+   "pick": "travel_advance_3",
+   "pick_run": "travel_advance_3",
+   "same": true,
+   "max_abs_dp": 0.0012396228532791098
+  },
+  {
+   "set": "nimble-eval",
+   "id": "nimble-eval:fast-41-diverse-009-004-counterfactual",
+   "qid": "decision",
+   "type": "noul",
+   "pick": "false",
+   "pick_run": "false",
+   "same": true,
+   "max_abs_dp": 0.0009295430221557544
+  },
+  {
+   "set": "nimble-eval",
+   "id": "nimble-eval:fast-43-diverse-174-010-counterfactual",
+   "qid": "decision",
+   "type": "score",
+   "pick": "0",
+   "pick_run": "0",
+   "same": true,
+   "max_abs_dp": 0.0011149642276763738
+  },
+  {
+   "set": "nimble-eval",
+   "id": "nimble-eval:fast-43-diverse-174-006-counterfactual",
+   "qid": "decision",
+   "type": "score",
+   "pick": "1",
+   "pick_run": "1",
+   "same": true,
+   "max_abs_dp": 0.0016253944072723292
+  },
+  {
+   "set": "nimble-eval",
+   "id": "nimble-eval:fast-41-diverse-009-001-base",
+   "qid": "decision",
+   "type": "noul",
+   "pick": "true",
+   "pick_run": "true",
+   "same": true,
+   "max_abs_dp": 0.00025924222946172026
+  },
+  {
+   "set": "nimble-eval",
+   "id": "nimble-eval:fast-43-diverse-079-058-base",
+   "qid": "decision",
+   "type": "choice",
+   "pick": "Proceed \u2014 High; no remaining routing",
+   "pick_run": "Proceed \u2014 High; no remaining routing",
+   "same": true,
+   "max_abs_dp": 0.0026827917366027965
+  },
+  {
+   "set": "nimble-eval",
+   "id": "nimble-eval:fast-43-diverse-171-016-base",
+   "qid": "decision",
+   "type": "noul",
+   "pick": "true",
+   "pick_run": "true",
+   "same": true,
+   "max_abs_dp": 3.662064181268222e-05
+  },
+  {
+   "set": "nimble-eval",
+   "id": "nimble-eval:scale-diverse-007-004-base",
+   "qid": "decision",
+   "type": "choice",
+   "pick": "major_seller_correction_not_ready",
+   "pick_run": "major_seller_correction_not_ready",
+   "same": true,
+   "max_abs_dp": 0.001442355716705368
+  },
+  {
+   "set": "nimble-eval",
+   "id": "nimble-eval:scale-diverse-171-001-base",
+   "qid": "decision",
+   "type": "noul",
+   "pick": "true",
+   "pick_run": "true",
+   "same": true,
+   "max_abs_dp": 0.0028766922264099604
+  },
+  {
+   "set": "nimble-eval",
+   "id": "nimble-eval:scale-diverse-275-004-counterfactual",
+   "qid": "decision",
+   "type": "score",
+   "pick": "0",
+   "pick_run": "0",
+   "same": true,
+   "max_abs_dp": 0.002890404029846194
+  },
+  {
+   "set": "nimble-eval",
+   "id": "nimble-eval:fast-43-diverse-169-053-counterfactual",
+   "qid": "decision",
+   "type": "choice",
+   "pick": "deny_low",
+   "pick_run": "deny_low",
+   "same": true,
+   "max_abs_dp": 0.004763497699737496
+  },
+  {
+   "set": "nimble-eval",
+   "id": "nimble-eval:fast-41-diverse-079-003-base",
+   "qid": "decision",
+   "type": "choice",
+   "pick": "Proceed \u2014 High; no remaining routing",
+   "pick_run": "Proceed \u2014 High; no remaining routing",
+   "same": true,
+   "max_abs_dp": 0.0015070434989929438
+  },
+  {
+   "set": "nimble-eval",
+   "id": "nimble-eval:fast-43-diverse-171-013-base",
+   "qid": "decision",
+   "type": "noul",
+   "pick": "true",
+   "pick_run": "true",
+   "same": true,
+   "max_abs_dp": 0.00013033839035037076
+  },
+  {
+   "set": "nimble-eval",
+   "id": "nimble-eval:fast-43-diverse-079-015-counterfactual",
+   "qid": "decision",
+   "type": "choice",
+   "pick": "Proceed \u2014 High; no remaining routing",
+   "pick_run": "Proceed \u2014 High; no remaining routing",
+   "same": true,
+   "max_abs_dp": 0.0023361680927276862
+  }
+ ]
+}

scripts/decide_standalone.py

@@ -26,6 +26,9 @@ def main():
     ap.add_argument("--attn", default="sdpa")
     a = ap.parse_args()
     req = json.load(open(a.request))
+    if a.device.startswith("mps"):
+        # transformers 5.17's threaded weight loader intermittently segfaults moving tensors to MPS
+        os.environ.setdefault("HF_DEACTIVATE_ASYNC_LOAD", "1")
     dtype = torch.bfloat16 if a.device.startswith(("cuda", "mps")) else torch.float32
     model_dir = a.model
     if not os.path.isdir(model_dir):