Taking Shortcuts for Categorical VQA Using Super Neurons
Sparse Attention Vectors (SAVs) have emerged as an excellent training-free alternative to supervised finetuning or low-rank adaptation to improve the performance of Vision Language Models (VLMs). At their heart, SAVs select a few accurate attention heads for a task of interest and use them as classifiers, rather than relying on the model's prediction. In a similar spirit, we find that directly probing the raw activations of the VLM, in the form of scalar values, is sufficient to yield accurate classifiers on diverse visually grounded downstream tasks. Shifting focus from attention vectors to scalar activations dramatically increases the search space for accurate parameters, allowing us to find more discriminative neurons immediately from the first generated token. We call such activations Super Neurons (SNs). In this probing setting, we discover that enough SNs appear in the shallower layers of the large language model to allow for extreme early exiting from the first layer of the model at the first generated token. Compared to the original network, SNs robustly improve the classification performance while achieving a speedup of up to 5.10x.
69b322642ad6c5d4cf486915
Pierre Musacchio
claimed_verified
2026-03-13T08:13:23.381Z
64a75d8374cc37096692593d
/avatars/295c1bbf9c3e8a13fbd3337c3ac0e86f.svg
Pierre Musacchio
pmusacchio
user
pmusacchio
69b322642ad6c5d4cf486916
Jaeyi Jeong
69b322642ad6c5d4cf486917
Dahun Kim
69b322642ad6c5d4cf486918
Jaesik Park
Taking Shortcuts for Categorical VQA Using Super Neurons
Sparse Attention Vectors (SAVs) have emerged as an excellent training-free alternative to supervised finetuning or low-rank adaptation to improve the performance of Vision Language Models (VLMs). At their heart, SAVs select a few accurate attention heads for a task of interest and use them as classifiers, rather than relying on the model's prediction. In a similar spirit, we find that directly probing the raw activations of the VLM, in the form of scalar values, is sufficient to yield accurate classifiers on diverse visually grounded downstream tasks. Shifting focus from attention vectors to scalar activations dramatically increases the search space for accurate parameters, allowing us to find more discriminative neurons immediately from the first generated token. We call such activations Super Neurons (SNs). In this probing setting, we discover that enough SNs appear in the shallower layers of the large language model to allow for extreme early exiting from the first layer of the model at the first generated token. Compared to the original network, SNs robustly improve the classification performance while achieving a speedup of up to 5.10x.
Sparse Attention Vectors (SAVs) are replaced by scalar activations called Super Neurons (SNs) that enable faster, more efficient classification in Vision-Language Models through early exiting from shallow layers.
Sparse Attention Vectors Vision Language Models attention heads classifiers scalar activations Super Neurons early exiting visual grounding downstream tasks