Focal Guidance: Unlocking Controllability from Semantic-Weak Layers in Video Diffusion Models
The task of Image-to-Video (I2V) generation aims to synthesize a video from a reference image and a text prompt. This requires diffusion models to reconcile high-frequency visual constraints and low-frequency textual guidance during the denoising process. However, while existing I2V models prioritize visual consistency, how to effectively couple this dual guidance to ensure strong adherence to the text prompt remains underexplored. In this work, we observe that in Diffusion Transformer (DiT)-based I2V models, certain intermediate layers exhibit weak semantic responses (termed Semantic-Weak Layers), as indicated by a measurable drop in text-visual similarity. We attribute this to a phenomenon called Condition Isolation, where attention to visual features becomes partially detached from text guidance and overly relies on learned visual priors. To address this, we propose Focal Guidance (FG), which enhances the controllability from Semantic-Weak Layers. FG comprises two mechanisms: (1) Fine-grained Semantic Guidance (FSG) leverages CLIP to identify key regions in the reference frame and uses them as anchors to guide Semantic-Weak Layers. (2) Attention Cache transfers attention maps from semantically responsive layers to Semantic-Weak Layers, injecting explicit semantic signals and alleviating their over-reliance on the model's learned visual priors, thereby enhancing adherence to textual instructions. To further validate our approach and address the lack of evaluation in this direction, we introduce a benchmark for assessing instruction following in I2V models. On this benchmark, Focal Guidance proves its effectiveness and generalizability, raising the total score on Wan2.1-I2V to 0.7250 (+3.97\%) and boosting the MMDiT-based HunyuanVideo-I2V to 0.5571 (+7.44\%).
69671bdac5e371f6b235d1c4
Yuanyang Yin
claimed_verified
2026-03-06T13:58:14.646Z
659cb2671d398a2381625b2f
/avatars/31ab5bf53fea8c9e6907aa3c921cc6ac.svg
SII-YuanyangYin
SII-YuanyangYin
user
SII-YuanyangYin
69671bdac5e371f6b235d1c5
Yufan Deng
claimed_verified
2026-01-14T09:51:08.659Z
68fce03ed1d0efce7ca87075
https://cdn-avatars.huggingface.co/v1/production/uploads/68fce03ed1d0efce7ca87075/GRKTeVIaLZD_M-KoJE8YF.png
yfdeng
yfdeng10
user
yfdeng10
69671bdac5e371f6b235d1c6
Shenghai Yuan
claimed_verified
2026-01-21T09:25:26.850Z
63468720dd6d90d82ccf3450
https://cdn-avatars.huggingface.co/v1/production/uploads/63468720dd6d90d82ccf3450/tVBFlmZNz8FRMkOrDaDID.jpeg
YSH
BestWishYsh
user
BestWishYsh
69671bdac5e371f6b235d1c7
Kaipeng Zhang
claimed_verified
2026-03-25T10:52:09.854Z
65f1713552c38a91e0a445e8
/avatars/47ab3ada51c9b9976ac1cd0c4301c373.svg
kaipeng
kpzhang996
user
kpzhang996
69671bdac5e371f6b235d1c8
Xiao Yang
claimed_verified
2026-03-05T09:58:42.654Z
62b3d2490385d6048985b42b
/avatars/1c4e534d086d53ef1057fbc00334350a.svg
Xiao Yang
seravee008
user
seravee008
69671bdac5e371f6b235d1c9
Feng Zhao
Focal Guidance: Unlocking Controllability from Semantic-Weak Layers in Video Diffusion Models
The task of Image-to-Video (I2V) generation aims to synthesize a video from a reference image and a text prompt. This requires diffusion models to reconcile high-frequency visual constraints and low-frequency textual guidance during the denoising process. However, while existing I2V models prioritize visual consistency, how to effectively couple this dual guidance to ensure strong adherence to the text prompt remains underexplored. In this work, we observe that in Diffusion Transformer (DiT)-based I2V models, certain intermediate layers exhibit weak semantic responses (termed Semantic-Weak Layers), as indicated by a measurable drop in text-visual similarity. We attribute this to a phenomenon called Condition Isolation, where attention to visual features becomes partially detached from text guidance and overly relies on learned visual priors. To address this, we propose Focal Guidance (FG), which enhances the controllability from Semantic-Weak Layers. FG comprises two mechanisms: (1) Fine-grained Semantic Guidance (FSG) leverages CLIP to identify key regions in the reference frame and uses them as anchors to guide Semantic-Weak Layers. (2) Attention Cache transfers attention maps from semantically responsive layers to Semantic-Weak Layers, injecting explicit semantic signals and alleviating their over-reliance on the model's learned visual priors, thereby enhancing adherence to textual instructions. To further validate our approach and address the lack of evaluation in this direction, we introduce a benchmark for assessing instruction following in I2V models. On this benchmark, Focal Guidance proves its effectiveness and generalizability, raising the total score on Wan2.1-I2V to 0.7250 (+3.97\%) and boosting the MMDiT-based HunyuanVideo-I2V to 0.5571 (+7.44\%).
Diffusion Transformer-based image-to-video models suffer from condition isolation where visual attention becomes detached from text guidance; focal guidance addresses this through fine-grained semantic guidance and attention cache mechanisms.
diffusion models diffusion transformer image-to-video generation text prompt denoising process semantic responses condition isolation attention mechanisms clip fine-grained semantic guidance attention cache