Can Large Language Models Keep Up? Benchmarking Online Adaptation to Continual Knowledge Streams
LLMs operating in dynamic real-world contexts often encounter knowledge that evolves continuously or emerges incrementally. To remain accurate and effective, models must adapt to newly arriving information on the fly. We introduce Online Adaptation to Continual Knowledge Streams(OAKS) to evaluate this capability, establishing a benchmark for online adaptation over streaming, continually updating knowledge. Specifically, the benchmark is structured as a sequence of fine-grained context chunks where facts change dynamically across time intervals. OAKS comprises two datasets: OAKS-BABI and OAKS-Novel, where individual facts evolve multiple times across context chunks. These datasets include dense annotations to measure whether models track changes accurately. Evaluating 14 models with varied inference approaches, we observe significant limitations in current methodologies. Both state-of-the-art models and agentic memory systems fail to adapt robustly on OAKS, demonstrating delays in state-tracking and susceptibility to distraction within streaming environments.
69afa5c2b3ab31dee1281e77
Jiyeon Kim
claimed_verified
2026-03-10T09:44:03.154Z
649003b36f54a65882ec370e
/avatars/4a47559a95f4a16cb22001d773070935.svg
Jiyeon Kim
jiyeonkim
user
jiyeonkim
69afa5c2b3ab31dee1281e78
Hyunji Lee
69afa5c2b3ab31dee1281e79
Dylan Zhou
69afa5c2b3ab31dee1281e7a
Sue Hyun Park
69afa5c2b3ab31dee1281e7b
Seunghyun Yoon
69afa5c2b3ab31dee1281e7c
Trung Bui
69afa5c2b3ab31dee1281e7d
Franck Dernoncourt
claimed_verified
2026-03-10T18:02:26.511Z
62c5947524171688a9feb992
/avatars/5a151713b9eae8dc566f5957acee3475.svg
Franck Dernoncourt
Franck-Dernoncourt
user
Franck-Dernoncourt
69afa5c2b3ab31dee1281e7e
Sungmin Cha
admin_assigned
2026-03-12T09:31:17.177Z
67b8df58c37862ea37f64a29
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/h6cxPTtyFkwlnteC45pxf.png
Sungmin Cha
csm9493
user
csm9493
69afa5c2b3ab31dee1281e7f
Minjoon Seo
admin_assigned
2026-03-12T09:31:24.253Z
621f05ba970615ad5861ceb1
/avatars/7e1902aa71369a524afda9b0a9e88e22.svg
Minjoon Seo
minjoon
user
minjoon
Can Large Language Models Keep Up? Benchmarking Online Adaptation to Continual Knowledge Streams
LLMs operating in dynamic real-world contexts often encounter knowledge that evolves continuously or emerges incrementally. To remain accurate and effective, models must adapt to newly arriving information on the fly. We introduce Online Adaptation to Continual Knowledge Streams(OAKS) to evaluate this capability, establishing a benchmark for online adaptation over streaming, continually updating knowledge. Specifically, the benchmark is structured as a sequence of fine-grained context chunks where facts change dynamically across time intervals. OAKS comprises two datasets: OAKS-BABI and OAKS-Novel, where individual facts evolve multiple times across context chunks. These datasets include dense annotations to measure whether models track changes accurately. Evaluating 14 models with varied inference approaches, we observe significant limitations in current methodologies. Both state-of-the-art models and agentic memory systems fail to adapt robustly on OAKS, demonstrating delays in state-tracking and susceptibility to distraction within streaming environments.
Online Adaptation to Continual Knowledge Streams (OAKS) benchmark evaluates how well language models adapt to dynamically changing information in real-time contexts through streaming datasets with evolving facts.
online adaptation continual knowledge streams streaming datasets dynamic real-world contexts state-of-the-art models agentic memory systems state-tracking distraction