BEAVER: An Efficient Deterministic LLM Verifier
As large language models (LLMs) transition from research prototypes to production systems, practitioners often need reliable methods to verify that model outputs satisfy required constraints. While sampling-based estimates provide an intuition of model behavior, they offer no sound guarantees. We present BEAVER, the first practical framework for computing deterministic, sound probability bounds on LLM constraint satisfaction. Given any prefix-closed semantic constraint, BEAVER systematically explores the generation space using novel token trie and frontier data structures, maintaining provably sound bounds at every iteration. We formalize the verification problem, prove soundness of our approach, and evaluate BEAVER on correctness verification, privacy verification and secure code generation tasks across multiple state of the art LLMs. BEAVER achieves 6 to 8 times tighter probability bounds and identifies 3 to 4 times more high risk instances compared to baseline methods under identical computational budgets, enabling precise characterization and risk assessment that loose bounds or empirical evaluation cannot provide.
693c5619f516c69324680c76
Tarun Suresh
claimed_verified
2026-04-15T12:10:32.630Z
65e7bb35e5e78134ab049942
/avatars/3c0972f0d59e51ebb5c218ee736d4458.svg
Tarun Suresh
tarsur909
user
tarsur909
693c5619f516c69324680c77
Nalin Wadhwa
693c5619f516c69324680c78
Debangshu Banerjee
claimed_verified
2026-02-09T08:37:34.081Z
675f9306e0b5cb5bc241e8cc
/avatars/566385d2049b353ea38ddbb9f9105dd0.svg
Debangshu Banerjee
debangshubanerjee
user
debangshubanerjee
693c5619f516c69324680c79
Gagandeep Singh
BEAVER: An Efficient Deterministic LLM Verifier
As large language models (LLMs) transition from research prototypes to production systems, practitioners often need reliable methods to verify that model outputs satisfy required constraints. While sampling-based estimates provide an intuition of model behavior, they offer no sound guarantees. We present BEAVER, the first practical framework for computing deterministic, sound probability bounds on LLM constraint satisfaction. Given any prefix-closed semantic constraint, BEAVER systematically explores the generation space using novel token trie and frontier data structures, maintaining provably sound bounds at every iteration. We formalize the verification problem, prove soundness of our approach, and evaluate BEAVER on correctness verification, privacy verification and secure code generation tasks across multiple state of the art LLMs. BEAVER achieves 6 to 8 times tighter probability bounds and identifies 3 to 4 times more high risk instances compared to baseline methods under identical computational budgets, enabling precise characterization and risk assessment that loose bounds or empirical evaluation cannot provide.
BEAVER is a framework that provides deterministic and sound probability bounds for verifying constraints in large language models, achieving tighter bounds and identifying more high-risk instances than baseline methods.
large language models LLMs BEAVER prefix-closed semantic constraint token trie frontier data structures sound probability bounds correctness verification privacy verification secure code generation high-risk instances