From Features to Actions: Explainability in Traditional and Agentic AI Systems
Over the last decade, explainable AI has primarily focused on interpreting individual model predictions, producing post-hoc explanations that relate inputs to outputs under a fixed decision structure. Recent advances in large language models (LLMs) have enabled agentic AI systems whose behaviour unfolds over multi-step trajectories. In these settings, success and failure are determined by sequences of decisions rather than a single output. While useful, it remains unclear how explanation approaches designed for static predictions translate to agentic settings where behaviour emerges over time. In this work, we bridge the gap between static and agentic explainability by comparing attribution-based explanations with trace-based diagnostics across both settings. To make this distinction explicit, we empirically compare attribution-based explanations used in static classification tasks with trace-based diagnostics used in agentic benchmarks (TAU-bench Airline and AssistantBench). Our results show that while attribution methods achieve stable feature rankings in static settings (Spearman ρ= 0.86), they cannot be applied reliably to diagnose execution-level failures in agentic trajectories. In contrast, trace-grounded rubric evaluation for agentic settings consistently localizes behaviour breakdowns and reveals that state tracking inconsistency is 2.7times more prevalent in failed runs and reduces success probability by 49\%. These findings motivate a shift towards trajectory-level explainability for agentic systems when evaluating and diagnosing autonomous AI behaviour.
  Resources:
  https://github.com/VectorInstitute/unified-xai-evaluation-framework https://vectorinstitute.github.io/unified-xai-evaluation-framework
698d0b7a6c5152984e4f3e37
Sindhuja Chaduvula
claimed_verified
2026-02-12T13:57:14.905Z
64c85111f3d2a59a43133747
/avatars/1fe304fc78553c4ac1a5704b7010aa0e.svg
Sindhuja Chaduvula
Sindhujach217
user
Sindhujach217
698d0b7a6c5152984e4f3e38
Jessee Ho
698d0b7a6c5152984e4f3e39
Kina Kim
698d0b7a6c5152984e4f3e3a
Aravind Narayanan
698d0b7a6c5152984e4f3e3b
Mahshid Alinoori
698d0b7a6c5152984e4f3e3c
Muskan Garg
698d0b7a6c5152984e4f3e3d
Dhanesh Ramachandram
698d0b7a6c5152984e4f3e3e
Shaina Raza
From Features to Actions: Explainability in Traditional and Agentic AI Systems
Over the last decade, explainable AI has primarily focused on interpreting individual model predictions, producing post-hoc explanations that relate inputs to outputs under a fixed decision structure. Recent advances in large language models (LLMs) have enabled agentic AI systems whose behaviour unfolds over multi-step trajectories. In these settings, success and failure are determined by sequences of decisions rather than a single output. While useful, it remains unclear how explanation approaches designed for static predictions translate to agentic settings where behaviour emerges over time. In this work, we bridge the gap between static and agentic explainability by comparing attribution-based explanations with trace-based diagnostics across both settings. To make this distinction explicit, we empirically compare attribution-based explanations used in static classification tasks with trace-based diagnostics used in agentic benchmarks (TAU-bench Airline and AssistantBench). Our results show that while attribution methods achieve stable feature rankings in static settings (Spearman ρ= 0.86), they cannot be applied reliably to diagnose execution-level failures in agentic trajectories. In contrast, trace-grounded rubric evaluation for agentic settings consistently localizes behaviour breakdowns and reveals that state tracking inconsistency is 2.7times more prevalent in failed runs and reduces success probability by 49\%. These findings motivate a shift towards trajectory-level explainability for agentic systems when evaluating and diagnosing autonomous AI behaviour.
  Resources:
  https://github.com/VectorInstitute/unified-xai-evaluation-framework https://vectorinstitute.github.io/unified-xai-evaluation-framework
Static and agentic explainability approaches differ in their ability to interpret model behavior, with attribution methods effective for individual predictions but inadequate for diagnosing failures in multi-step decision processes, where trace-based diagnostics prove more reliable.
explainable AI large language models agentic AI post-hoc explanations attribution-based explanations trace-based diagnostics static predictions agentic trajectories TAU-bench AssistantBench trajectory-level explainability