RAPTOR: Ridge-Adaptive Logistic Probes
Probing studies what information is encoded in a frozen LLM's layer representations by training a lightweight predictor on top of them. Beyond analysis, probes are often used operationally in probe-then-steer pipelines: a learned concept vector is extracted from a probe and injected via additive activation steering by adding it to a layer representation during the forward pass. The effectiveness of this pipeline hinges on estimating concept vectors that are accurate, directionally stable under ablation, and inexpensive to obtain. Motivated by these desiderata, we propose RAPTOR (Ridge-Adaptive Logistic Probe), a simple L2-regularized logistic probe whose validation-tuned ridge strength yields concept vectors from normalized weights. Across extensive experiments on instruction-tuned LLMs and human-written concept datasets, RAPTOR matches or exceeds strong baselines in accuracy while achieving competitive directional stability and substantially lower training cost; these quantitative results are supported by qualitative downstream steering demonstrations. Finally, using the Convex Gaussian Min-max Theorem (CGMT), we provide a mechanistic characterization of ridge logistic regression in an idealized Gaussian teacher-student model in the high-dimensional few-shot regime, explaining how penalty strength mediates probe accuracy and concept-vector stability and yielding structural predictions that qualitatively align with trends observed on real LLM embeddings.
69815a13ce18b18628095f4c
Ziqi Gao
claimed_verified
2026-02-03T10:08:05.332Z
67f2d22288d089a03d678250
/avatars/7f6b5df381acefd0c02fec509a75baf5.svg
Ziqi Gao
Ziqi-Gao
user
Ziqi-Gao
69815a13ce18b18628095f4d
Yaotian Zhu
69815a13ce18b18628095f4e
Qingcheng Zeng
69815a13ce18b18628095f4f
Xu Zhao
claimed_verified
2026-02-03T13:59:44.006Z
6782f626952fd17459daada2
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/rJkGWnoEuKQl3UIC32nYA.png
Xu Zhao
XuZhao2025
user
XuZhao2025
69815a13ce18b18628095f50
Ziqing Wang
69815a13ce18b18628095f51
Feng Ruan
69815a13ce18b18628095f52
Kaize Ding
RAPTOR: Ridge-Adaptive Logistic Probes
Probing studies what information is encoded in a frozen LLM's layer representations by training a lightweight predictor on top of them. Beyond analysis, probes are often used operationally in probe-then-steer pipelines: a learned concept vector is extracted from a probe and injected via additive activation steering by adding it to a layer representation during the forward pass. The effectiveness of this pipeline hinges on estimating concept vectors that are accurate, directionally stable under ablation, and inexpensive to obtain. Motivated by these desiderata, we propose RAPTOR (Ridge-Adaptive Logistic Probe), a simple L2-regularized logistic probe whose validation-tuned ridge strength yields concept vectors from normalized weights. Across extensive experiments on instruction-tuned LLMs and human-written concept datasets, RAPTOR matches or exceeds strong baselines in accuracy while achieving competitive directional stability and substantially lower training cost; these quantitative results are supported by qualitative downstream steering demonstrations. Finally, using the Convex Gaussian Min-max Theorem (CGMT), we provide a mechanistic characterization of ridge logistic regression in an idealized Gaussian teacher-student model in the high-dimensional few-shot regime, explaining how penalty strength mediates probe accuracy and concept-vector stability and yielding structural predictions that qualitatively align with trends observed on real LLM embeddings.
RACTOR, a ridge-adaptive logistic probe, achieves accurate and stable concept vector estimation for activation steering in frozen LLMs with reduced training costs, supported by theoretical analysis of ridge logistic regression in high-dimensional settings.
LLM probes activation steering concept vectors ridge regularization logistic regression teacher-student model CGMT high-dimensional regime few-shot learning