Benchmarking Knowledge-Extraction Attack and Defense on Retrieval-Augmented Generation
Retrieval-Augmented Generation (RAG) has become a cornerstone of knowledge-intensive applications, including enterprise chatbots, healthcare assistants, and agentic memory management. However, recent studies show that knowledge-extraction attacks can recover sensitive knowledge-base content through maliciously crafted queries, raising serious concerns about intellectual property theft and privacy leakage. While prior work has explored individual attack and defense techniques, the research landscape remains fragmented, spanning heterogeneous retrieval embeddings, diverse generation models, and evaluations based on non-standardized metrics and inconsistent datasets. To address this gap, we introduce the first systematic benchmark for knowledge-extraction attacks on RAG systems. Our benchmark covers a broad spectrum of attack and defense strategies, representative retrieval embedding models, and both open- and closed-source generators, all evaluated under a unified experimental framework with standardized protocols across multiple datasets. By consolidating the experimental landscape and enabling reproducible, comparable evaluation, this benchmark provides actionable insights and a practical foundation for developing privacy-preserving RAG systems in the face of emerging knowledge extraction threats. Our code is available here.
6994359350fb2c0be4783e87
Zhisheng Qi
6994359350fb2c0be4783e88
Utkarsh Sahu
6994359350fb2c0be4783e89
Li Ma
6994359350fb2c0be4783e8a
Haoyu Han
6994359350fb2c0be4783e8b
Ryan Rossi
6994359350fb2c0be4783e8c
Franck Dernoncourt
claimed_verified
2026-02-17T15:08:43.351Z
62c5947524171688a9feb992
/avatars/5a151713b9eae8dc566f5957acee3475.svg
Franck Dernoncourt
Franck-Dernoncourt
user
Franck-Dernoncourt
6994359350fb2c0be4783e8d
Mahantesh Halappanavar
6994359350fb2c0be4783e8e
Nesreen Ahmed
6994359350fb2c0be4783e8f
Yushun Dong
6994359350fb2c0be4783e90
Yue Zhao
6994359350fb2c0be4783e91
Yu Zhang
6994359350fb2c0be4783e92
Yu Wang
Benchmarking Knowledge-Extraction Attack and Defense on Retrieval-Augmented Generation
Retrieval-Augmented Generation (RAG) has become a cornerstone of knowledge-intensive applications, including enterprise chatbots, healthcare assistants, and agentic memory management. However, recent studies show that knowledge-extraction attacks can recover sensitive knowledge-base content through maliciously crafted queries, raising serious concerns about intellectual property theft and privacy leakage. While prior work has explored individual attack and defense techniques, the research landscape remains fragmented, spanning heterogeneous retrieval embeddings, diverse generation models, and evaluations based on non-standardized metrics and inconsistent datasets. To address this gap, we introduce the first systematic benchmark for knowledge-extraction attacks on RAG systems. Our benchmark covers a broad spectrum of attack and defense strategies, representative retrieval embedding models, and both open- and closed-source generators, all evaluated under a unified experimental framework with standardized protocols across multiple datasets. By consolidating the experimental landscape and enabling reproducible, comparable evaluation, this benchmark provides actionable insights and a practical foundation for developing privacy-preserving RAG systems in the face of emerging knowledge extraction threats. Our code is available here.
A systematic benchmark for evaluating knowledge-extraction attacks on Retrieval-Augmented Generation systems is introduced, covering diverse attack and defense strategies across multiple retrieval and generation models with standardized evaluation protocols.
Retrieval-Augmented Generation knowledge-extraction attacks attack and defense strategies retrieval embedding models generation models standardized protocols reproducible evaluation