Query-focused and Memory-aware Reranker for Long Context Processing
Built upon the existing analysis of retrieval heads in large language models, we propose an alternative reranking framework that trains models to estimate passage-query relevance using the attention scores of selected heads. This approach provides a listwise solution that leverages holistic information within the entire candidate shortlist during ranking. At the same time, it naturally produces continuous relevance scores, enabling training on arbitrary retrieval datasets without requiring Likert-scale supervision. Our framework is lightweight and effective, requiring only small-scale models (e.g., 4B parameters) to achieve strong performance. Extensive experiments demonstrate that our method outperforms existing state-of-the-art pointwise and listwise rerankers across multiple domains, including Wikipedia and long narrative datasets. It further establishes a new state-of-the-art on the LoCoMo benchmark that assesses the capabilities of dialogue understanding and memory usage. We further demonstrate that our framework supports flexible extensions. For example, augmenting candidate passages with contextual information further improves ranking accuracy, while training attention heads from middle layers enhances efficiency without sacrificing performance.
698ebfdacace060ff123aece
Yuqing Li
claimed_verified
2026-02-13T09:36:07.802Z
6891bbff78946201296b4592
https://cdn-avatars.huggingface.co/v1/production/uploads/6891bbff78946201296b4592/ECmWBrlfeonPg0HzmQ_sW.png
Yuqing Li
MindscapeRAG
user
MindscapeRAG
698ebfdacace060ff123aecf
Jiangnan Li
claimed_verified
2026-02-26T10:03:50.952Z
6670f52a7a0446ec70e11c96
/avatars/7786b9d947d7bbaf95013feee3a3bf39.svg
Jiangnan Li
lossisnotanumber
user
lossisnotanumber
698ebfdacace060ff123aed0
Mo Yu
claimed_verified
2026-02-25T17:32:43.530Z
67af92045a86287292026808
/avatars/8bad9272fe73ba04e077b5484837c8d3.svg
Mo
BishopGorov
user
BishopGorov
698ebfdacace060ff123aed1
Guoxuan Ding
698ebfdacace060ff123aed2
Zheng Lin
698ebfdacace060ff123aed3
Weiping Wang
698ebfdacace060ff123aed4
Jie Zhou
Query-focused and Memory-aware Reranker for Long Context Processing
Built upon the existing analysis of retrieval heads in large language models, we propose an alternative reranking framework that trains models to estimate passage-query relevance using the attention scores of selected heads. This approach provides a listwise solution that leverages holistic information within the entire candidate shortlist during ranking. At the same time, it naturally produces continuous relevance scores, enabling training on arbitrary retrieval datasets without requiring Likert-scale supervision. Our framework is lightweight and effective, requiring only small-scale models (e.g., 4B parameters) to achieve strong performance. Extensive experiments demonstrate that our method outperforms existing state-of-the-art pointwise and listwise rerankers across multiple domains, including Wikipedia and long narrative datasets. It further establishes a new state-of-the-art on the LoCoMo benchmark that assesses the capabilities of dialogue understanding and memory usage. We further demonstrate that our framework supports flexible extensions. For example, augmenting candidate passages with contextual information further improves ranking accuracy, while training attention heads from middle layers enhances efficiency without sacrificing performance.
A lightweight reranking framework uses attention scores from selected heads to estimate passage-query relevance, achieving strong performance across multiple domains and benchmarks.
retrieval heads reranking framework attention scores passage-query relevance listwise solution candidate shortlist continuous relevance scores Likert-scale supervision pointwise rerankers listwise rerankers LoCoMo benchmark dialogue understanding memory usage contextual information middle layers