Layer by layer, module by module: Choose both for optimal OOD probing of ViT
Recent studies have observed that intermediate layers of foundation models often yield more discriminative representations than the final layer. While initially attributed to autoregressive pretraining, this phenomenon has also been identified in models trained via supervised and discriminative self-supervised objectives. In this paper, we conduct a comprehensive study to analyze the behavior of intermediate layers in pretrained vision transformers. Through extensive linear probing experiments across a diverse set of image classification benchmarks, we find that distribution shift between pretraining and downstream data is the primary cause of performance degradation in deeper layers. Furthermore, we perform a fine-grained analysis at the module level. Our findings reveal that standard probing of transformer block outputs is suboptimal; instead, probing the activation within the feedforward network yields the best performance under significant distribution shift, whereas the normalized output of the multi-head self-attention module is optimal when the shift is weak.
69ae8edd9893f58272f7424f
Ambroise Odonnat
claimed_verified
2026-03-09T09:39:33.619Z
6661f79ac36ae4c83f3213e4
/avatars/2e48f052fdf37b5b06d101a6a3232eea.svg
Ambroise Odonnat
ambroiseodt
user
ambroiseodt
69ae8edd9893f58272f74250
Vasilii Feofanov
69ae8edd9893f58272f74251
Laetitia Chapel
69ae8edd9893f58272f74252
Romain Tavenard
69ae8edd9893f58272f74253
Ievgen Redko
Layer by layer, module by module: Choose both for optimal OOD probing of ViT
Recent studies have observed that intermediate layers of foundation models often yield more discriminative representations than the final layer. While initially attributed to autoregressive pretraining, this phenomenon has also been identified in models trained via supervised and discriminative self-supervised objectives. In this paper, we conduct a comprehensive study to analyze the behavior of intermediate layers in pretrained vision transformers. Through extensive linear probing experiments across a diverse set of image classification benchmarks, we find that distribution shift between pretraining and downstream data is the primary cause of performance degradation in deeper layers. Furthermore, we perform a fine-grained analysis at the module level. Our findings reveal that standard probing of transformer block outputs is suboptimal; instead, probing the activation within the feedforward network yields the best performance under significant distribution shift, whereas the normalized output of the multi-head self-attention module is optimal when the shift is weak.
Intermediate layers in vision transformers provide better representations than final layers, with performance degradation due to distribution shifts between pretraining and downstream data, and optimal probing strategies depend on the magnitude of these shifts.
vision transformers intermediate layers final layer autoregressive pretraining supervised learning discriminative self-supervised objectives linear probing distribution shift transformer block feedforward network multi-head self-attention module