Causal Concept Graphs in LLM Latent Space for Stepwise Reasoning
Sparse autoencoders can localize where concepts live in language models, but not how they interact during multi-step reasoning. We propose Causal Concept Graphs (CCG): a directed acyclic graph over sparse, interpretable latent features, where edges capture learned causal dependencies between concepts. We combine task-conditioned sparse autoencoders for concept discovery with DAGMA-style differentiable structure learning for graph recovery and introduce the Causal Fidelity Score (CFS) to evaluate whether graph-guided interventions induce larger downstream effects than random ones. On ARC-Challenge, StrategyQA, and LogiQA with GPT-2 Medium, across five seeds (n{=}15 paired runs), CCG achieves CFS=5.654pm0.625, outperforming ROME-style tracing (3.382pm0.233), SAE-only ranking (2.479pm0.196), and a random baseline (1.032pm0.034), with p<0.0001 after Bonferroni correction. Learned graphs are sparse (5-6\% edge density), domain-specific, and stable across seeds.
69b2c3d07450df3b5c089ceb
Md Muntaqim Meherab
69b2c3d07450df3b5c089cec
Noor Islam S. Mohammad
claimed_verified
2026-03-13T08:14:01.551Z
66fc350c4fb09ea16665645d
https://cdn-avatars.huggingface.co/v1/production/uploads/66fc350c4fb09ea16665645d/xuUdgtj9pZ_bJ3zcK0t0J.jpeg
Noor Islam S. Mohammad
nislamsm
user
nislamsm
69b2c3d07450df3b5c089ced
Faiza Feroz
Causal Concept Graphs in LLM Latent Space for Stepwise Reasoning
Sparse autoencoders can localize where concepts live in language models, but not how they interact during multi-step reasoning. We propose Causal Concept Graphs (CCG): a directed acyclic graph over sparse, interpretable latent features, where edges capture learned causal dependencies between concepts. We combine task-conditioned sparse autoencoders for concept discovery with DAGMA-style differentiable structure learning for graph recovery and introduce the Causal Fidelity Score (CFS) to evaluate whether graph-guided interventions induce larger downstream effects than random ones. On ARC-Challenge, StrategyQA, and LogiQA with GPT-2 Medium, across five seeds (n{=}15 paired runs), CCG achieves CFS=5.654pm0.625, outperforming ROME-style tracing (3.382pm0.233), SAE-only ranking (2.479pm0.196), and a random baseline (1.032pm0.034), with p<0.0001 after Bonferroni correction. Learned graphs are sparse (5-6\% edge density), domain-specific, and stable across seeds.
Causal Concept Graphs leverage sparse autoencoders and differentiable structure learning to identify causal relationships between concepts in language models, demonstrating superior causal fidelity compared to alternative methods.
sparse autoencoders causal dependencies directed acyclic graph concept discovery DAGMA-style structure learning causal fidelity score downstream effects graph-guided interventions language models