Legal RAG Bench: an end-to-end benchmark for legal RAG
We introduce Legal RAG Bench, a benchmark and evaluation methodology for assessing the end-to-end performance of legal RAG systems. As a benchmark, Legal RAG Bench consists of 4,876 passages from the Victorian Criminal Charge Book alongside 100 complex, hand-crafted questions demanding expert knowledge of criminal law and procedure. Both long-form answers and supporting passages are provided. As an evaluation methodology, Legal RAG Bench leverages a full factorial design and novel hierarchical error decomposition framework, enabling apples-to-apples comparisons of the contributions of retrieval and reasoning models in RAG. We evaluate three state-of-the-art embedding models (Isaacus' Kanon 2 Embedder, Google's Gemini Embedding 001, and OpenAI's Text Embedding 3 Large) and two frontier LLMs (Gemini 3.1 Pro and GPT-5.2), finding that information retrieval is the primary driver of legal RAG performance, with LLMs exerting a more moderate effect on correctness and groundedness. Kanon 2 Embedder, in particular, had the largest positive impact on performance, improving average correctness by 17.5 points, groundedness by 4.5 points, and retrieval accuracy by 34 points. We observe that many errors attributed to hallucinations in legal RAG systems are in fact triggered by retrieval failures, concluding that retrieval sets the ceiling for the performance of many modern legal RAG systems. We document why and how we built Legal RAG Bench alongside the results of our evaluations. We also openly release our code and data to assist with reproduction of our findings.
69a65311663a122a7ce666e1
Abdur-Rahman Butler
claimed_verified
2026-03-03T10:39:30.207Z
63568098ef1d4c9191533b3f
https://cdn-avatars.huggingface.co/v1/production/uploads/63568098ef1d4c9191533b3f/o9IGAr_XBBiMtGRGeO3V3.png
Abdur-Rahman Butler
abdurrahmanbutler
user
abdurrahmanbutler
69a65311663a122a7ce666e2
Umar Butler
claimed_verified
2026-03-03T10:39:37.276Z
6497ffbf2a997a45e987e139
https://cdn-avatars.huggingface.co/v1/production/uploads/6497ffbf2a997a45e987e139/LUipMEC-wC1QeIXHSMnsx.png
Umar Butler
umarbutler
user
umarbutler
Legal RAG Bench: an end-to-end benchmark for legal RAG
We introduce Legal RAG Bench, a benchmark and evaluation methodology for assessing the end-to-end performance of legal RAG systems. As a benchmark, Legal RAG Bench consists of 4,876 passages from the Victorian Criminal Charge Book alongside 100 complex, hand-crafted questions demanding expert knowledge of criminal law and procedure. Both long-form answers and supporting passages are provided. As an evaluation methodology, Legal RAG Bench leverages a full factorial design and novel hierarchical error decomposition framework, enabling apples-to-apples comparisons of the contributions of retrieval and reasoning models in RAG. We evaluate three state-of-the-art embedding models (Isaacus' Kanon 2 Embedder, Google's Gemini Embedding 001, and OpenAI's Text Embedding 3 Large) and two frontier LLMs (Gemini 3.1 Pro and GPT-5.2), finding that information retrieval is the primary driver of legal RAG performance, with LLMs exerting a more moderate effect on correctness and groundedness. Kanon 2 Embedder, in particular, had the largest positive impact on performance, improving average correctness by 17.5 points, groundedness by 4.5 points, and retrieval accuracy by 34 points. We observe that many errors attributed to hallucinations in legal RAG systems are in fact triggered by retrieval failures, concluding that retrieval sets the ceiling for the performance of many modern legal RAG systems. We document why and how we built Legal RAG Bench alongside the results of our evaluations. We also openly release our code and data to assist with reproduction of our findings.
Legal RAG Bench evaluates legal retrieval-augmented generation systems using a comprehensive dataset and factorial analysis, revealing that information retrieval significantly impacts performance more than language model capabilities.
legal RAG retrieval-augmented generation embedding models large language models information retrieval hallucinations factorial design hierarchical error decomposition