Spectral Attention Steering for Prompt Highlighting
Attention steering is an important technique for controlling model focus, enabling capabilities such as prompt highlighting, where the model prioritises user-specified text. However, existing attention steering methods require explicit storage of the full attention matrix, making them incompatible with memory-efficient implementations like FlashAttention. We introduce Spectral Editing Key Amplification (SEKA), a training-free steering method that tackles this by directly editing key embeddings before attention computation. SEKA uses spectral decomposition to steer key embeddings towards latent directions that amplify attention scores for certain tokens. We extend this to Adaptive SEKA (AdaSEKA), a query-adaptive variant that uses a training-free routing mechanism to dynamically combine multiple expert subspaces based on the prompt's semantic intent. Our experiments show both methods significantly outperform strong baselines on standard steering benchmarks while adding much lower latency and memory overhead, in compatibility with optimised attention.
69a69d97663a122a7ce6696b
Weixian Waylon Li
69a69d97663a122a7ce6696c
Yuchen Niu
69a69d97663a122a7ce6696d
Yongxin Yang
69a69d97663a122a7ce6696e
Keshuang Li
69a69d97663a122a7ce6696f
Tiejun Ma
69a69d97663a122a7ce66970
Shay B. Cohen
Spectral Attention Steering for Prompt Highlighting
Attention steering is an important technique for controlling model focus, enabling capabilities such as prompt highlighting, where the model prioritises user-specified text. However, existing attention steering methods require explicit storage of the full attention matrix, making them incompatible with memory-efficient implementations like FlashAttention. We introduce Spectral Editing Key Amplification (SEKA), a training-free steering method that tackles this by directly editing key embeddings before attention computation. SEKA uses spectral decomposition to steer key embeddings towards latent directions that amplify attention scores for certain tokens. We extend this to Adaptive SEKA (AdaSEKA), a query-adaptive variant that uses a training-free routing mechanism to dynamically combine multiple expert subspaces based on the prompt's semantic intent. Our experiments show both methods significantly outperform strong baselines on standard steering benchmarks while adding much lower latency and memory overhead, in compatibility with optimised attention.
Spectral Editing Key Amplification (SEKA) and its adaptive variant AdaSEKA enable efficient attention steering by modifying key embeddings before attention computation, achieving superior performance with reduced memory overhead compared to traditional methods.
attention steering FlashAttention spectral decomposition key embeddings attention scores spectral editing key amplification adaptive SEKA query-adaptive expert subspaces prompt semantic intent