FVG-PT: Adaptive Foreground View-Guided Prompt Tuning for Vision-Language Models
CLIP-based prompt tuning enables pretrained Vision-Language Models (VLMs) to efficiently adapt to downstream tasks. Although existing studies have made significant progress, they pay limited attention to changes in the internal attention representations of VLMs during the tuning process. In this paper, we attribute the failure modes of prompt tuning predictions to shifts in foreground attention of the visual encoder, and propose Foreground View-Guided Prompt Tuning (FVG-PT), an adaptive plug-and-play foreground attention guidance module, to alleviate the shifts. Concretely, FVG-PT introduces a learnable Foreground Reliability Gate to automatically enhance the foreground view quality, applies a Foreground Distillation Compensation module to guide visual attention toward the foreground, and further introduces a Prior Calibration module to mitigate generalization degradation caused by excessive focus on the foreground. Experiments on multiple backbone models and datasets show the effectiveness and compatibility of FVG-PT. Codes are available at: https://github.com/JREion/FVG-PT
69af8be9b3ab31dee1281c97
Haoyang Li
claimed_verified
2026-03-10T09:45:01.657Z
645372e74871691ae74cab6b
https://cdn-avatars.huggingface.co/v1/production/uploads/645372e74871691ae74cab6b/UFMRQJDpdQU6kuVLuT-bW.png
Haoyang Li
JREion
user
JREion
69af8be9b3ab31dee1281c98
Liang Wang
69af8be9b3ab31dee1281c99
Siyu Zhou
69af8be9b3ab31dee1281c9a
Jiacheng Sun
69af8be9b3ab31dee1281c9b
Jing Jiang
69af8be9b3ab31dee1281c9c
Chao Wang
69af8be9b3ab31dee1281c9d
Guodong Long
69af8be9b3ab31dee1281c9e
Yan Peng
FVG-PT: Adaptive Foreground View-Guided Prompt Tuning for Vision-Language Models
CLIP-based prompt tuning enables pretrained Vision-Language Models (VLMs) to efficiently adapt to downstream tasks. Although existing studies have made significant progress, they pay limited attention to changes in the internal attention representations of VLMs during the tuning process. In this paper, we attribute the failure modes of prompt tuning predictions to shifts in foreground attention of the visual encoder, and propose Foreground View-Guided Prompt Tuning (FVG-PT), an adaptive plug-and-play foreground attention guidance module, to alleviate the shifts. Concretely, FVG-PT introduces a learnable Foreground Reliability Gate to automatically enhance the foreground view quality, applies a Foreground Distillation Compensation module to guide visual attention toward the foreground, and further introduces a Prior Calibration module to mitigate generalization degradation caused by excessive focus on the foreground. Experiments on multiple backbone models and datasets show the effectiveness and compatibility of FVG-PT. Codes are available at: https://github.com/JREion/FVG-PT
Foreground attention shifts during CLIP-based prompt tuning are addressed through an adaptive module that enhances foreground view quality and mitigates generalization degradation.
prompt tuning Vision-Language Models visual encoder foreground attention Foreground Reliability Gate Foreground Distillation Compensation Prior Calibration