AthenaBench: A Dynamic Benchmark for Evaluating LLMs in Cyber Threat
  Intelligence
Large Language Models (LLMs) have demonstrated strong capabilities in natural
language reasoning, yet their application to Cyber Threat Intelligence (CTI)
remains limited. CTI analysis involves distilling large volumes of unstructured
reports into actionable knowledge, a process where LLMs could substantially
reduce analyst workload. CTIBench introduced a comprehensive benchmark for
evaluating LLMs across multiple CTI tasks. In this work, we extend CTIBench by
developing AthenaBench, an enhanced benchmark that includes an improved dataset
creation pipeline, duplicate removal, refined evaluation metrics, and a new
task focused on risk mitigation strategies. We evaluate twelve LLMs, including
state-of-the-art proprietary models such as GPT-5 and Gemini-2.5 Pro, alongside
seven open-source models from the LLaMA and Qwen families. While proprietary
LLMs achieve stronger results overall, their performance remains subpar on
reasoning-intensive tasks, such as threat actor attribution and risk
mitigation, with open-source models trailing even further behind. These
findings highlight fundamental limitations in the reasoning capabilities of
current LLMs and underscore the need for models explicitly tailored to CTI
workflows and automation.
690a90ebd70e173c84528f36
Md Tanvirul Alam
claimed_verified
2025-11-17T10:32:13.750Z
620c1a977af55d45f5519914
/avatars/4d2fb7c1a5ad5dabdb8888fa2fe72e65.svg
Tanvirul Alam
xashru
user
xashru
690a90ebd70e173c84528f37
Dipkamal Bhusal
690a90ebd70e173c84528f38
Salman Ahmad
690a90ebd70e173c84528f39
Nidhi Rastogi
690a90ebd70e173c84528f3a
Peter Worth
AthenaBench: A Dynamic Benchmark for Evaluating LLMs in Cyber Threat
  Intelligence
Large Language Models (LLMs) have demonstrated strong capabilities in natural
language reasoning, yet their application to Cyber Threat Intelligence (CTI)
remains limited. CTI analysis involves distilling large volumes of unstructured
reports into actionable knowledge, a process where LLMs could substantially
reduce analyst workload. CTIBench introduced a comprehensive benchmark for
evaluating LLMs across multiple CTI tasks. In this work, we extend CTIBench by
developing AthenaBench, an enhanced benchmark that includes an improved dataset
creation pipeline, duplicate removal, refined evaluation metrics, and a new
task focused on risk mitigation strategies. We evaluate twelve LLMs, including
state-of-the-art proprietary models such as GPT-5 and Gemini-2.5 Pro, alongside
seven open-source models from the LLaMA and Qwen families. While proprietary
LLMs achieve stronger results overall, their performance remains subpar on
reasoning-intensive tasks, such as threat actor attribution and risk
mitigation, with open-source models trailing even further behind. These
findings highlight fundamental limitations in the reasoning capabilities of
current LLMs and underscore the need for models explicitly tailored to CTI
workflows and automation.
AthenaBench, an enhanced benchmark for evaluating LLMs in CTI, reveals limitations in reasoning capabilities of current models, especially for tasks like threat actor attribution and risk mitigation.
Large Language Models CTI Cyber Threat Intelligence CTIBench AthenaBench dataset creation pipeline duplicate removal evaluation metrics risk mitigation strategies GPT-5 Gemini-2.5 Pro LLaMA Qwen