Text Generation
MLX
Safetensors
English
Chinese
qwen3_5
apus-reflex
decision-model
apple-silicon
conversational
4-bit precision
Instructions to use apus-ailab/APUS-Reflex-4B-MLX-4bit with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- MLX
How to use apus-ailab/APUS-Reflex-4B-MLX-4bit with MLX:
# Make sure mlx-lm is installed # pip install --upgrade mlx-lm # Generate text with mlx-lm from mlx_lm import load, generate model, tokenizer = load("apus-ailab/APUS-Reflex-4B-MLX-4bit") prompt = "Write a story about Einstein" messages = [{"role": "user", "content": prompt}] prompt = tokenizer.apply_chat_template( messages, add_generation_prompt=True ) text = generate(model, tokenizer, prompt=prompt, verbose=True) - Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- LM Studio
- Pi
How to use apus-ailab/APUS-Reflex-4B-MLX-4bit with Pi:
Start the MLX server
# Install MLX LM: uv tool install mlx-lm # Start a local OpenAI-compatible server: mlx_lm.server --model "apus-ailab/APUS-Reflex-4B-MLX-4bit"
Configure the model in Pi
# Install Pi: npm install -g @earendil-works/pi-coding-agent # Add to ~/.pi/agent/models.json: { "providers": { "mlx-lm": { "baseUrl": "http://localhost:8080/v1", "api": "openai-completions", "apiKey": "none", "models": [ { "id": "apus-ailab/APUS-Reflex-4B-MLX-4bit" } ] } } }Run Pi
# Start Pi in your project directory: pi
- MLX LM
How to use apus-ailab/APUS-Reflex-4B-MLX-4bit with MLX LM:
Generate or start a chat session
# Install MLX LM uv tool install mlx-lm # Interactive chat REPL mlx_lm.chat --model "apus-ailab/APUS-Reflex-4B-MLX-4bit"
Run an OpenAI-compatible server
# Install MLX LM uv tool install mlx-lm # Start the server mlx_lm.server --model "apus-ailab/APUS-Reflex-4B-MLX-4bit" # Calling the OpenAI-compatible server with curl curl -X POST "http://localhost:8080/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "apus-ailab/APUS-Reflex-4B-MLX-4bit", "messages": [ {"role": "user", "content": "Hello"} ] }' - Hermes Agent
How to use apus-ailab/APUS-Reflex-4B-MLX-4bit with Hermes Agent:
Start the MLX server
# Install MLX LM: uv tool install mlx-lm # Start a local OpenAI-compatible server: mlx_lm.server --model "apus-ailab/APUS-Reflex-4B-MLX-4bit"
Configure Hermes
# Install Hermes: curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash hermes setup # Point Hermes at the local server: hermes config set model.provider custom hermes config set model.base_url http://127.0.0.1:8080/v1 hermes config set model.default apus-ailab/APUS-Reflex-4B-MLX-4bit
Run Hermes
hermes
- Atomic Chat
- OpenClaw
How to use apus-ailab/APUS-Reflex-4B-MLX-4bit with OpenClaw:
Start the MLX server
# Install MLX LM: uv tool install mlx-lm # Start a local OpenAI-compatible server: mlx_lm.server --model "apus-ailab/APUS-Reflex-4B-MLX-4bit"
Configure OpenClaw
# Install OpenClaw: npm install -g openclaw@latest # Register the local server and set it as the default model: openclaw onboard --non-interactive --mode local \ --auth-choice custom-api-key \ --custom-base-url http://127.0.0.1:8080/v1 \ --custom-model-id "apus-ailab/APUS-Reflex-4B-MLX-4bit" \ --custom-provider-id mlx-lm \ --custom-compatibility openai \ --custom-text-input \ --accept-risk \ --skip-health
Run OpenClaw
openclaw agent --local --agent main --message "Hello from Hugging Face"
Download evaluation/per-question/mlx-4bit-mac.jsonl from apus-ailab/APUS-Reflex-4B-MLX-4bit: direct link, hf CLI and curl.
- Browser
- Download file 27.8 kB
-
https://huggingface.co/apus-ailab/APUS-Reflex-4B-MLX-4bit/resolve/main/evaluation/per-question/mlx-4bit-mac.jsonl
- Command line
-
hf download hf://apus-ailab/APUS-Reflex-4B-MLX-4bit/evaluation/per-question/mlx-4bit-mac.jsonl
-
curl -L -o mlx-4bit-mac.jsonl https://huggingface.co/apus-ailab/APUS-Reflex-4B-MLX-4bit/resolve/main/evaluation/per-question/mlx-4bit-mac.jsonl
27.8 kB
| {"panel_index": 0, "task_family": "browser", "logits": [13.0, 13.125, 12.25, 12.5625, 20.5, 12.1875, 12.125, 12.25, 12.9375, 13.375, 11.9375, 11.125, 11.9375, 12.1875, 11.25, 11.75], "probabilities": [0.000550427648704499, 0.0006237163324840367, 0.0002600035513751209, 0.00035538262454792857, 0.9951962828636169, 0.0002442508121021092, 0.00022945231467019767, 0.0002600035513751209, 0.000517078791745007, 0.000800867157522589, 0.00019022270862478763, 8.441080717602745e-05, 0.00019022270862478763, 0.0002442508121021092, 9.56499861786142e-05, 0.00015770010941196233], "choice": "candidate-c59cf6b3cdd1ee8dfe92", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 0.001094043254852295} | |
| {"panel_index": 1, "task_family": "browser", "logits": [13.25, 13.4375, 12.5625, 12.875, 12.5625, 12.3125, 12.3125, 12.6875, 20.375, 16.625, 13.0, 12.75, 12.75, 13.0625, 12.0625, 12.625], "probabilities": [0.0007807706715539098, 0.00094178959261626, 0.00039259615004993975, 0.0005366155528463423, 0.00039259615004993975, 0.00030575436539947987, 0.00030575436539947987, 0.0004448697727639228, 0.9702234268188477, 0.022817471995949745, 0.000608065165579319, 0.00047356158029288054, 0.00047356158029288054, 0.0006472818204201758, 0.00023812172003090382, 0.0004179165989626199], "choice": "candidate-d2449a13158601200ec0", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 0.007025726139545441} | |
| {"panel_index": 2, "task_family": "browser", "logits": [13.25, 13.5625, 12.875, 13.5625, 14.0, 20.125, 12.875, 13.25, 13.375, 12.0, 12.25, 11.875, 12.9375, 12.75, 11.4375, 12.625], "probabilities": [0.0010203273268416524, 0.001394622609950602, 0.0007012603455223143, 0.001394622609950602, 0.002160033443942666, 0.9874476790428162, 0.0007012603455223143, 0.0010203273268416524, 0.0011561823775991797, 0.0002923287102021277, 0.00037535751471295953, 0.00025797917624004185, 0.0007464875816367567, 0.0006188600673340261, 0.00016656388470437378, 0.0005461420514620841], "choice": "candidate-9409b72a36db8d4a0a79", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 0.004585385322570801} | |
| {"panel_index": 3, "task_family": "browser", "logits": [15.75, 14.5625, 13.625, 13.9375, 14.5625, 14.25, 18.75, 14.3125, 14.5, 14.625, 18.0, 12.875, 13.6875, 17.75, 18.75, 14.0], "probabilities": [0.016564618796110153, 0.005051922518759966, 0.0019783612806349993, 0.0027040995191782713, 0.005051922518759966, 0.0036960665602236986, 0.3327092230319977, 0.0039344425313174725, 0.004745841957628727, 0.005377743858844042, 0.1571607142686844, 0.000934512005187571, 0.0021059545688331127, 0.12239688634872437, 0.3327092230319977, 0.0028784992173314095], "choice": "candidate-3b25ae8d064a298ef021", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 0.5350258052349091} | |
| {"panel_index": 4, "task_family": "browser", "logits": [20.5, 13.6875, 12.875, 13.8125, 12.375, 12.0, 11.5625, 11.625, 12.125, 10.8125, 11.5625, 11.25, 12.125, 12.0625, 11.125, 12.0625], "probabilities": [0.9951499104499817, 0.0010946050751954317, 0.0004857280873693526, 0.0012403501896187663, 0.0002946089080069214, 0.0002024815184995532, 0.00013073190348222852, 0.00013916344323661178, 0.00022944161901250482, 6.175340240588412e-05, 0.00013073190348222852, 9.564553329255432e-05, 0.00022944161901250482, 0.00021554052364081144, 8.440687815891579e-05, 0.00021554052364081144], "choice": "candidate-34e89f28b090ce5327b0", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 0.0007718801498413086} | |
| {"panel_index": 5, "task_family": "browser", "logits": [20.0, 13.3125, 12.75, 12.5, 13.6875, 12.3125, 11.6875, 15.5625, 12.5625, 11.1875, 11.6875, 11.3125, 11.9375, 12.8125, 12.3125, 11.875], "probabilities": [0.9805555939674377, 0.0012221598299220204, 0.0006963657215237617, 0.0005423301481641829, 0.0017782311188057065, 0.0004496073233895004, 0.00024065753677859902, 0.01159552950412035, 0.0005773072480224073, 0.0001459661580156535, 0.00024065753677859902, 0.00016540133219677955, 0.0003090104437433183, 0.0007412772974930704, 0.0004496073233895004, 0.00029028832796029747], "choice": "candidate-fb3025ec54721504f9b5", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 0.0036859222454950213} | |
| {"panel_index": 6, "task_family": "browser", "logits": [13.375, 13.4375, 12.4375, 12.5, 13.375, 13.3125, 13.9375, 14.875, 13.375, 16.125, 19.5, 12.0625, 12.625, 13.0625, 11.4375, 13.75], "probabilities": [0.002049564616754651, 0.0021817495580762625, 0.0008026210125535727, 0.0008543853182345629, 0.002049564616754651, 0.0019253870705142617, 0.003597097471356392, 0.00918551068753004, 0.002049564616754651, 0.03206057474017143, 0.9369474053382874, 0.0005516327219083905, 0.0009681455558165908, 0.0014994937228038907, 0.0002952678478322923, 0.0029820993077009916], "choice": "candidate-888bb5997d1aba7b25f7", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 0.007417528890073299} | |
| {"panel_index": 7, "task_family": "browser", "logits": [20.125, 13.0, 11.9375, 12.0, 12.1875, 11.25, 11.1875, 11.3125, 11.875, 10.375, 11.3125, 10.8125, 12.0, 11.625, 10.4375, 11.3125], "probabilities": [0.9965865612030029, 0.0008019859669730067, 0.00027715909527614713, 0.00029503420228138566, 0.00035587910679169, 0.00013936434697825462, 0.00013092064182274044, 0.0001483525411458686, 0.0002603667671792209, 5.8095683925785124e-05, 0.0001483525411458686, 8.998034172691405e-05, 0.00029503420228138566, 0.0002027738228207454, 6.18425547145307e-05, 0.0001483525411458686], "choice": "candidate-41c5af16d7bf373cc74c", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 0.0005077719688415527} | |
| {"panel_index": 8, "task_family": "browser", "logits": [12.5, 12.6875, 20.25, 13.25, 12.5625, 11.75, 13.3125, 11.9375, 12.125, 10.9375, 11.25, 10.9375, 12.0, 11.5625, 10.75, 11.625], "probabilities": [0.000428589090006426, 0.0005169770447537303, 0.995000422000885, 0.0009073228575289249, 0.0004562305985018611, 0.00020245110499672592, 0.0009658404742367566, 0.00024420273257419467, 0.0002945646410807967, 8.983712905319408e-05, 0.0001227928587468341, 8.983712905319408e-05, 0.00025995238684117794, 0.00016783789033070207, 7.447761890944093e-05, 0.0001786624634405598], "choice": "candidate-a1a3ad1a8495aa678cbe", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 0.0006424784660339355} | |
| {"panel_index": 9, "task_family": "browser", "logits": [19.75, 15.375, 13.25, 13.125, 13.0625, 14.5625, 12.1875, 12.9375, 13.0625, 11.9375, 12.375, 11.9375, 12.5625, 12.75, 11.5, 12.4375], "probabilities": [0.9716766476631165, 0.012231603264808655, 0.001460856874473393, 0.001289201551117003, 0.0012110931565985084, 0.005427740979939699, 0.0005048585589975119, 0.001068785903044045, 0.0012110931565985084, 0.00039318419294431806, 0.000608975940849632, 0.00039318419294431806, 0.000734565022867173, 0.0008860543021000922, 0.0002538588596507907, 0.0006482513272203505], "choice": "candidate-976c2731e7518a1f1b2b", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 0.007759273052215576} | |
| {"panel_index": 10, "task_family": "browser", "logits": [14.8125, 17.5, 13.0625, 13.875, 16.75, 14.75, 13.625, 14.75, 14.5, 14.125, 14.3125, 18.75, 14.0625, 17.125, 12.5, 13.6875], "probabilities": [0.011171646416187286, 0.1641661375761032, 0.0019413409754633904, 0.004374879412353039, 0.07754659652709961, 0.010494790971279144, 0.003407159121707082, 0.010494790971279144, 0.008173350244760513, 0.0056174565106630325, 0.006775946822017431, 0.5729960799217224, 0.005277112126350403, 0.11282961815595627, 0.0011061428813263774, 0.0036269021220505238], "choice": "candidate-016db9380de74cbcab5d", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 0.2969970703125} | |
| {"panel_index": 11, "task_family": "browser", "logits": [16.625, 14.0625, 13.1875, 13.25, 13.3125, 12.625, 12.5, 12.9375, 13.5, 16.625, 13.3125, 13.0, 15.5, 19.5, 12.0, 12.875], "probabilities": [0.04894805699586868, 0.0037744687870144844, 0.0015734320040792227, 0.0016749103087931871, 0.0017829322023317218, 0.0008965147426351905, 0.0007911713910289109, 0.0012253908207640052, 0.0021506277844309807, 0.04894805699586868, 0.0017829322023317218, 0.001304421341046691, 0.01589110679924488, 0.8676248788833618, 0.0004798698937520385, 0.0011511476477608085], "choice": "candidate-2dc16f49fe3acb051b39", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 0.056685686111450195} | |
| {"panel_index": 12, "task_family": "browser", "logits": [13.5, 13.3125, 13.1875, 13.8125, 12.5625, 12.8125, 12.375, 12.625, 14.25, 12.375, 12.4375, 12.375, 13.5625, 17.625, 11.8125, 20.375], "probabilities": [0.000961938058026135, 0.0007974748150445521, 0.0007037690375000238, 0.001314813969656825, 0.0003767003072425723, 0.00048369282740168273, 0.0003122956259176135, 0.00040099554462358356, 0.002036423422396183, 0.0003122956259176135, 0.0003324368444737047, 0.0003122956259176135, 0.0010239779949188232, 0.05951300635933876, 0.0001779406884452328, 0.9309399724006653], "choice": "candidate-bb1945d8d438379625ee", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 0.03607302904129028} | |
| {"panel_index": 13, "task_family": "browser", "logits": [13.5, 13.0, 11.875, 12.1875, 11.6875, 11.875, 12.8125, 12.75, 12.625, 12.625, 18.625, 11.625, 13.25, 13.5, 15.125, 13.3125], "probabilities": [0.005547683220356703, 0.00336484145373106, 0.0010924035450443625, 0.0014931393088772893, 0.0009056346607394516, 0.0010924035450443625, 0.0027895504608750343, 0.002620541024953127, 0.002312619239091873, 0.002312619239091873, 0.9329769015312195, 0.0008507646853104234, 0.004320540931075811, 0.005547683220356703, 0.028173459693789482, 0.004599192179739475], "choice": "candidate-94dddd06eb43ed2921b0", "correct": false, "agrees_with_reference": true, "max_abs_probability_delta": 0.21123641729354858} | |
| {"panel_index": 14, "task_family": "browser", "logits": [13.0, 20.125, 12.25, 12.3125, 11.875, 11.5, 11.6875, 11.9375, 12.3125, 11.5625, 11.8125, 14.75, 12.4375, 11.8125, 11.375, 12.25], "probabilities": [0.0007973653264343739, 0.9908447861671448, 0.00037664888077415526, 0.0004009404801763594, 0.00025886669754981995, 0.0001779162703314796, 0.00021460808056872338, 0.0002755622554104775, 0.0004009404801763594, 0.00018939095025416464, 0.0002431828179396689, 0.004588522482663393, 0.0004543251416180283, 0.0002431828179396689, 0.00015701055235695094, 0.00037664888077415526], "choice": "candidate-b56ad12ceea4c52feb3f", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 0.006659030914306641} | |
| {"panel_index": 15, "task_family": "browser", "logits": [17.75, 17.0, 15.0625, 19.625, 17.75, 15.4375, 14.375, 17.375, 14.5625, 14.875, 16.0, 14.0, 15.6875, 14.8125, 12.5625, 13.4375], "probabilities": [0.09637756645679474, 0.04552553966641426, 0.006558576133102179, 0.6284605860710144, 0.09637756645679474, 0.009542670100927353, 0.003297859337180853, 0.06623926758766174, 0.003977976739406586, 0.0054372502490878105, 0.016747910529375076, 0.0022665837313979864, 0.012253032997250557, 0.005107823759317398, 0.000538360676728189, 0.0012914597755298018], "choice": "candidate-da095abe0a63c03f3114", "correct": false, "agrees_with_reference": true, "max_abs_probability_delta": 0.07026156038045883} | |
| {"panel_index": 16, "task_family": "hs3", "logits": [14.6875, 18.75], "probabilities": [0.016914915293455124, 0.9830850958824158], "choice": "no", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 0.025172829627990723} | |
| {"panel_index": 17, "task_family": "hs3", "logits": [15.6875, 18.75], "probabilities": [0.0446808785200119, 0.9553191661834717], "choice": "no", "correct": false, "agrees_with_reference": true, "max_abs_probability_delta": 0.2761404514312744} | |
| {"panel_index": 18, "task_family": "hs3", "logits": [19.625, 17.0], "probabilities": [0.9324532747268677, 0.06754669547080994], "choice": "yes", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 0.03021981194615364} | |
| {"panel_index": 19, "task_family": "hs3", "logits": [18.875, 13.0625], "probabilities": [0.9970190525054932, 0.0029810331761837006], "choice": "yes", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 0.0006578706670552492} | |
| {"panel_index": 20, "task_family": "hs3", "logits": [19.0, 13.1875], "probabilities": [0.9970190525054932, 0.0029810331761837006], "choice": "yes", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 0.0006578706670552492} | |
| {"panel_index": 21, "task_family": "hs3", "logits": [11.625, 18.375], "probabilities": [0.0011695101857185364, 0.9988304972648621], "choice": "no", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 7.077911868691444e-05} | |
| {"panel_index": 22, "task_family": "hs3", "logits": [19.25, 12.9375], "probabilities": [0.9981897473335266, 0.0018102105241268873], "choice": "yes", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 0.0006407499313354492} | |
| {"panel_index": 23, "task_family": "hs3", "logits": [19.0, 17.0], "probabilities": [0.8807970285415649, 0.11920292675495148], "choice": "yes", "correct": false, "agrees_with_reference": true, "max_abs_probability_delta": 0.09324556961655617} | |
| {"panel_index": 24, "task_family": "hs3", "logits": [19.125, 13.25], "probabilities": [0.9971990585327148, 0.0028009277302771807], "choice": "yes", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 0.0006182193756103516} | |
| {"panel_index": 25, "task_family": "hs3", "logits": [13.1875, 19.25], "probabilities": [0.0023231625091284513, 0.9976769089698792], "choice": "no", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 0.00047785043716430664} | |
| {"panel_index": 26, "task_family": "hs3", "logits": [17.75, 18.375], "probabilities": [0.3486451804637909, 0.6513549089431763], "choice": "no", "correct": true, "agrees_with_reference": false, "max_abs_probability_delta": 0.5979515761137009} | |
| {"panel_index": 27, "task_family": "hs3", "logits": [19.5, 13.5], "probabilities": [0.9975274205207825, 0.002472624182701111], "choice": "yes", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 0.0007719015702605247} | |
| {"panel_index": 28, "task_family": "hs3", "logits": [15.875, 18.875], "probabilities": [0.047425881028175354, 0.9525741338729858], "choice": "no", "correct": false, "agrees_with_reference": true, "max_abs_probability_delta": 0.009226560592651367} | |
| {"panel_index": 29, "task_family": "hs3", "logits": [12.125, 18.5], "probabilities": [0.001700722612440586, 0.9982993006706238], "choice": "no", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 0.00010287761688232422} | |
| {"panel_index": 30, "task_family": "hs3", "logits": [18.5, 15.25], "probabilities": [0.9626730680465698, 0.037326887249946594], "choice": "yes", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 0.02820134162902832} | |
| {"panel_index": 31, "task_family": "hs3", "logits": [19.5, 13.6875], "probabilities": [0.9970190525054932, 0.0029810331761837006], "choice": "yes", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 0.0010542984819039702} | |
| {"panel_index": 32, "task_family": "boolq", "logits": [13.3125, 18.75], "probabilities": [0.004331503063440323, 0.9956685304641724], "choice": "no", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 0.00027811527252197266} | |
| {"panel_index": 33, "task_family": "boolq", "logits": [12.25, 19.125], "probabilities": [0.0010322310263291001, 0.9989677667617798], "choice": "no", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 0.00046896934509277344} | |
| {"panel_index": 34, "task_family": "boolq", "logits": [12.625, 18.625], "probabilities": [0.002472624182701111, 0.9975274205207825], "choice": "no", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 0.0005084087606519461} | |
| {"panel_index": 35, "task_family": "boolq", "logits": [19.625, 13.25], "probabilities": [0.9982993006706238, 0.001700722612440586], "choice": "yes", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 0.0007719006389379501} | |
| {"panel_index": 36, "task_family": "boolq", "logits": [20.125, 12.4375], "probabilities": [0.9995417594909668, 0.0004583129193633795], "choice": "yes", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 2.775582834146917e-05} | |
| {"panel_index": 37, "task_family": "boolq", "logits": [20.125, 12.8125], "probabilities": [0.9993333220481873, 0.0006667023990303278], "choice": "yes", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 0.00020843744277954102} | |
| {"panel_index": 38, "task_family": "boolq", "logits": [19.75, 12.5], "probabilities": [0.9992903470993042, 0.0007096707122400403], "choice": "yes", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 8.338689804077148e-05} | |
| {"panel_index": 39, "task_family": "boolq", "logits": [13.0, 19.25], "probabilities": [0.001926734927110374, 0.9980732202529907], "choice": "no", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 0.0008741921046748757} | |
| {"panel_index": 40, "task_family": "boolq", "logits": [19.875, 12.5], "probabilities": [0.999373733997345, 0.0006263344548642635], "choice": "yes", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 0.00019577736384235322} | |
| {"panel_index": 41, "task_family": "boolq", "logits": [13.125, 18.875], "probabilities": [0.003172683995217085, 0.9968273043632507], "choice": "no", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 0.000897454097867012} | |
| {"panel_index": 42, "task_family": "boolq", "logits": [19.625, 12.5625], "probabilities": [0.9991440773010254, 0.0008559008128941059], "choice": "yes", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 0.0001462697982788086} | |
| {"panel_index": 43, "task_family": "boolq", "logits": [15.25, 18.375], "probabilities": [0.04208773374557495, 0.957912266254425], "choice": "no", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 0.014502525329589844} | |
| {"panel_index": 44, "task_family": "boolq", "logits": [19.125, 13.125], "probabilities": [0.9975274205207825, 0.002472624182701111], "choice": "yes", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 0.0007719015702605247} | |
| {"panel_index": 45, "task_family": "boolq", "logits": [20.625, 14.375], "probabilities": [0.9980732202529907, 0.001926734927110374], "choice": "yes", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 0.0007572770118713379} | |
| {"panel_index": 46, "task_family": "boolq", "logits": [13.75, 18.75], "probabilities": [0.006692850962281227, 0.9933071732521057], "choice": "no", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 0.0013692378997802734} | |
| {"panel_index": 47, "task_family": "boolq", "logits": [12.6875, 18.875], "probabilities": [0.0020507434383034706, 0.9979493021965027], "choice": "no", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 0.0005809664726257324} | |
| {"panel_index": 48, "task_family": "mnli", "logits": [16.125, 19.75, 14.125], "probabilities": [0.025866493582725525, 0.9706328511238098, 0.0035006501711905003], "choice": "neutral", "correct": false, "agrees_with_reference": true, "max_abs_probability_delta": 0.013438940048217773} | |
| {"panel_index": 49, "task_family": "mnli", "logits": [13.25, 20.75, 12.1875], "probabilities": [0.0005526732420548797, 0.9992563128471375, 0.0001909987477120012], "choice": "neutral", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 0.00017827749252319336} | |
| {"panel_index": 50, "task_family": "mnli", "logits": [20.5, 12.6875, 11.6875], "probabilities": [0.9994468092918396, 0.0004044212691951543, 0.00014877831563353539], "choice": "entailment", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 9.593830327503383e-06} | |
| {"panel_index": 51, "task_family": "mnli", "logits": [14.375, 20.25, 13.25], "probabilities": [0.0027983831241726875, 0.9962931275367737, 0.0009085016208700836], "choice": "neutral", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 0.002107858657836914} | |
| {"panel_index": 52, "task_family": "mnli", "logits": [11.4375, 12.3125, 20.0], "probabilities": [0.00019101679208688438, 0.0004582253168337047, 0.9993507266044617], "choice": "contradiction", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 5.382605013437569e-05} | |
| {"panel_index": 53, "task_family": "mnli", "logits": [20.5, 12.3125, 11.375], "probabilities": [0.9996131062507629, 0.0002780008071567863, 0.00010886666859732941], "choice": "entailment", "correct": false, "agrees_with_reference": true, "max_abs_probability_delta": 4.875659942626953e-05} | |
| {"panel_index": 54, "task_family": "mnli", "logits": [15.0, 20.0, 13.1875], "probabilities": [0.006685546599328518, 0.9922230839729309, 0.0010913857258856297], "choice": "neutral", "correct": false, "agrees_with_reference": true, "max_abs_probability_delta": 0.054238319396972656} | |
| {"panel_index": 55, "task_family": "mnli", "logits": [20.375, 12.6875, 11.625], "probabilities": [0.9993834495544434, 0.00045824030530638993, 0.00015836360398679972], "choice": "entailment", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 5.820766091346741e-11} | |
| {"panel_index": 56, "task_family": "mnli", "logits": [20.25, 12.5, 11.625], "probabilities": [0.9993901252746582, 0.0004304799367673695, 0.00017945068248081952], "choice": "entailment", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 7.867813110351562e-05} | |
| {"panel_index": 57, "task_family": "mnli", "logits": [14.625, 17.75, 19.125], "probabilities": [0.008789122104644775, 0.20003947615623474, 0.7911713719367981], "choice": "contradiction", "correct": false, "agrees_with_reference": true, "max_abs_probability_delta": 0.12851053476333618} | |
| {"panel_index": 58, "task_family": "mnli", "logits": [14.1875, 19.5, 17.25], "probabilities": [0.00443975580856204, 0.9006341099739075, 0.0949261486530304], "choice": "neutral", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 0.0856391191482544} | |
| {"panel_index": 59, "task_family": "mnli", "logits": [19.625, 14.25, 13.9375], "probabilities": [0.9920448064804077, 0.004594079684466124, 0.003361100098118186], "choice": "entailment", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 0.10517442226409912} | |
| {"panel_index": 60, "task_family": "mnli", "logits": [14.25, 15.6875, 19.75], "probabilities": [0.00400156807154417, 0.01684722863137722, 0.9791512489318848], "choice": "contradiction", "correct": false, "agrees_with_reference": true, "max_abs_probability_delta": 0.011214613914489746} | |
| {"panel_index": 61, "task_family": "mnli", "logits": [19.625, 15.75, 13.875], "probabilities": [0.976622462272644, 0.020269155502319336, 0.0031083764042705297], "choice": "entailment", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 0.040536701679229736} | |
| {"panel_index": 62, "task_family": "mnli", "logits": [20.125, 12.6875, 11.875], "probabilities": [0.9991506338119507, 0.0005882551777176559, 0.00026103665004484355], "choice": "entailment", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 9.495019912719727e-05} | |
| {"panel_index": 63, "task_family": "mnli", "logits": [20.375, 12.375, 11.5], "probabilities": [0.9995249509811401, 0.00033530330983921885, 0.00013977526396047324], "choice": "entailment", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 0.0001087188720703125} | |
| {"panel_index": 64, "task_family": "score", "logits": [19.5, 17.375], "probabilities": [0.8933094143867493, 0.10669060051441193], "choice": "yes", "correct": false, "agrees_with_reference": true, "max_abs_probability_delta": 0.30064283311367035} | |
| {"panel_index": 65, "task_family": "score", "logits": [12.0625, 19.75], "probabilities": [0.0004583129193633795, 0.9995417594909668], "choice": "no", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 2.956390380859375e-05} | |
| {"panel_index": 66, "task_family": "score", "logits": [12.1875, 20.0], "probabilities": [0.00040448145591653883, 0.999595582485199], "choice": "no", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 5.3831463446840644e-05} | |
| {"panel_index": 67, "task_family": "score", "logits": [13.25, 19.625], "probabilities": [0.001700722612440586, 0.9982993006706238], "choice": "no", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 0.0014719963073730469} | |
| {"panel_index": 68, "task_family": "score", "logits": [12.375, 19.625], "probabilities": [0.0007096707122400403, 0.9992903470993042], "choice": "no", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 0.00020138052059337497} | |
| {"panel_index": 69, "task_family": "score", "logits": [20.0, 12.8125], "probabilities": [0.9992445707321167, 0.0007554057519882917], "choice": "yes", "correct": false, "agrees_with_reference": true, "max_abs_probability_delta": 4.57763671875e-05} | |
| {"panel_index": 70, "task_family": "score", "logits": [12.9375, 19.75], "probabilities": [0.0010987314162775874, 0.9989012479782104], "choice": "no", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 0.0001289844512939453} | |
| {"panel_index": 71, "task_family": "score", "logits": [12.75, 19.75], "probabilities": [0.0009110511164180934, 0.9990890026092529], "choice": "no", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 0.00010696519166231155} | |
| {"panel_index": 72, "task_family": "score", "logits": [19.25, 18.0], "probabilities": [0.7772998809814453, 0.22270014882087708], "choice": "yes", "correct": false, "agrees_with_reference": true, "max_abs_probability_delta": 0.24609050154685974} | |
| {"panel_index": 73, "task_family": "score", "logits": [12.0, 19.625], "probabilities": [0.00048785729450173676, 0.9995121955871582], "choice": "no", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 3.153085708618164e-05} | |
| {"panel_index": 74, "task_family": "score", "logits": [12.25, 20.125], "probabilities": [0.0003799845580942929, 0.9996199607849121], "choice": "no", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 0.0001393202692270279} | |
| {"panel_index": 75, "task_family": "score", "logits": [12.75, 19.625], "probabilities": [0.0010322310263291001, 0.9989677667617798], "choice": "no", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 0.00021266937255859375} | |
| {"panel_index": 76, "task_family": "score", "logits": [17.75, 19.625], "probabilities": [0.13296425342559814, 0.8670357465744019], "choice": "no", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 0.11604934930801392} | |
| {"panel_index": 77, "task_family": "score", "logits": [14.25, 19.5], "probabilities": [0.0052201272919774055, 0.9947799444198608], "choice": "no", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 0.016395211219787598} | |
| {"panel_index": 78, "task_family": "score", "logits": [13.8125, 19.75], "probabilities": [0.0026316740550100803, 0.997368335723877], "choice": "no", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 0.0} | |
| {"panel_index": 79, "task_family": "score", "logits": [13.375, 19.5], "probabilities": [0.0021827169694006443, 0.9978172779083252], "choice": "no", "correct": true, "agrees_with_reference": true, "max_abs_probability_delta": 4.656612873077393e-10} | |