Semantic Richness or Geometric Reasoning? The Fragility of VLM's Visual Invariance
This work investigates the fundamental fragility of state-of-the-art Vision-Language Models (VLMs) under basic geometric transformations. While modern VLMs excel at semantic tasks such as recognizing objects in canonical orientations and describing complex scenes, they exhibit systematic failures at a more fundamental level: lack of robust spatial invariance and equivariance required to reliably determine object identity under simple rotations, scaling, and identity transformations. We demonstrate this limitation through a systematic evaluation across diverse visual domains, including symbolic sketches, natural photographs, and abstract art. Performance drops sharply as semantic content becomes sparse, and this behavior is observed across architectures, model capacities, and prompting strategies. Overall, our results reveal a systematic gap between semantic understanding and spatial reasoning in current VLMs, highlighting the need for stronger geometric grounding in future multimodal systems.
69d397e42c523d36431fae15
Jason Qiu
69d397e42c523d36431fae16
Zachary Meurer
69d397e42c523d36431fae17
Xavier Thomas
claimed_verified
2026-04-13T07:35:02.326Z
6505b787dacc94cd6ce9d256
/avatars/9490807f97afc39aeb663d343a900ff9.svg
Thomas
XThomasBU
user
XThomasBU
69d397e42c523d36431fae18
Deepti Ghadiyaram
claimed_verified
2026-04-13T07:35:04.343Z
6905068e4c9138632af2fa21
/avatars/c32215347601de938c218e6c81dc2e33.svg
Deepti Ghadiyaram
dghadiya
user
dghadiya
Semantic Richness or Geometric Reasoning? The Fragility of VLM's Visual Invariance
This work investigates the fundamental fragility of state-of-the-art Vision-Language Models (VLMs) under basic geometric transformations. While modern VLMs excel at semantic tasks such as recognizing objects in canonical orientations and describing complex scenes, they exhibit systematic failures at a more fundamental level: lack of robust spatial invariance and equivariance required to reliably determine object identity under simple rotations, scaling, and identity transformations. We demonstrate this limitation through a systematic evaluation across diverse visual domains, including symbolic sketches, natural photographs, and abstract art. Performance drops sharply as semantic content becomes sparse, and this behavior is observed across architectures, model capacities, and prompting strategies. Overall, our results reveal a systematic gap between semantic understanding and spatial reasoning in current VLMs, highlighting the need for stronger geometric grounding in future multimodal systems.
Vision-Language Models show significant vulnerabilities under geometric transformations, lacking robust spatial invariance and equivariance despite strong semantic capabilities.
Vision-Language Models geometric transformations spatial invariance spatial equivariance semantic understanding multimodal systems