Assessing LLMs for Serendipity Discovery in Knowledge Graphs: A Case for Drug Repurposing
Large Language Models (LLMs) have greatly advanced knowledge graph question answering (KGQA), yet existing systems are typically optimized for returning highly relevant but predictable answers. A missing yet desired capacity is to exploit LLMs to suggest surprise and novel ("serendipitious") answers. In this paper, we formally define the serendipity-aware KGQA task and propose the SerenQA framework to evaluate LLMs' ability to uncover unexpected insights in scientific KGQA tasks. SerenQA includes a rigorous serendipity metric based on relevance, novelty, and surprise, along with an expert-annotated benchmark derived from the Clinical Knowledge Graph, focused on drug repurposing. Additionally, it features a structured evaluation pipeline encompassing three subtasks: knowledge retrieval, subgraph reasoning, and serendipity exploration. Our experiments reveal that while state-of-the-art LLMs perform well on retrieval, they still struggle to identify genuinely surprising and valuable discoveries, underscoring a significant room for future improvements. Our curated resources and extended version are released at: https://cwru-db-group.github.io/serenQA.
691cf7d392ec6730e840e110
Mengying Wang
691cf7d392ec6730e840e111
Chenhui Ma
691cf7d392ec6730e840e112
Ao Jiao
691cf7d392ec6730e840e113
Tuo Liang
691cf7d392ec6730e840e114
Pengjun Lu
691cf7d392ec6730e840e115
Shrinidhi Hegde
691cf7d392ec6730e840e116
Yu Yin
691cf7d392ec6730e840e117
Evren Gurkan-Cavusoglu
691cf7d392ec6730e840e118
Yinghui Wu
Assessing LLMs for Serendipity Discovery in Knowledge Graphs: A Case for Drug Repurposing
Large Language Models (LLMs) have greatly advanced knowledge graph question answering (KGQA), yet existing systems are typically optimized for returning highly relevant but predictable answers. A missing yet desired capacity is to exploit LLMs to suggest surprise and novel ("serendipitious") answers. In this paper, we formally define the serendipity-aware KGQA task and propose the SerenQA framework to evaluate LLMs' ability to uncover unexpected insights in scientific KGQA tasks. SerenQA includes a rigorous serendipity metric based on relevance, novelty, and surprise, along with an expert-annotated benchmark derived from the Clinical Knowledge Graph, focused on drug repurposing. Additionally, it features a structured evaluation pipeline encompassing three subtasks: knowledge retrieval, subgraph reasoning, and serendipity exploration. Our experiments reveal that while state-of-the-art LLMs perform well on retrieval, they still struggle to identify genuinely surprising and valuable discoveries, underscoring a significant room for future improvements. Our curated resources and extended version are released at: https://cwru-db-group.github.io/serenQA.
SerenQA evaluates Large Language Models' ability to generate surprising and valuable answers in scientific knowledge graph question answering, particularly in drug repurposing, using a structured pipeline and serendipity metric.
Large Language Models LLMS knowledge graph question answering KGQA serendipity-aware KGQA SerenQA serendipity metric subgraph reasoning serendipity exploration drug repurposing knowledge retrieval