Specificity-aware reinforcement learning for fine-grained open-world classification
Classifying fine-grained visual concepts under open-world settings, i.e., without a predefined label set, demands models to be both accurate and specific. Recent reasoning Large Multimodal Models (LMMs) exhibit strong visual understanding capability but tend to produce overly generic predictions when performing fine-grained image classification. Our preliminary analysis reveals that models do possess the intrinsic fine-grained domain knowledge. However, promoting more specific predictions (specificity) without compromising correct ones (correctness) remains a non-trivial and understudied challenge. In this work, we investigate how to steer reasoning LMMs toward predictions that are both correct and specific. We propose a novel specificity-aware reinforcement learning framework, SpeciaRL, to fine-tune reasoning LMMs on fine-grained image classification under the open-world setting. SpeciaRL introduces a dynamic, verifier-based reward signal anchored to the best predictions within online rollouts, promoting specificity while respecting the model's capabilities to prevent incorrect predictions. Our out-of-domain experiments show that SpeciaRL delivers the best trade-off between correctness and specificity across extensive fine-grained benchmarks, surpassing existing methods and advancing open-world fine-grained image classification. Code and model are publicly available at https://github.com/s-angheben/SpeciaRL.
69a958ccddca602df39671f0
Samuele Angheben
claimed_verified
2026-03-05T11:01:15.659Z
696a065e7cf44ee3a5646042
https://cdn-avatars.huggingface.co/v1/production/uploads/696a065e7cf44ee3a5646042/c1QZAtEXzPfKkJ4cZ35st.jpeg
Samuele Angheben
s-angheben
user
s-angheben
69a958ccddca602df39671f1
Davide Berasi
claimed_verified
2026-03-09T10:38:03.844Z
6597e62f929cd840d808b8c9
/avatars/c3c074390d745b17488551d7ea0999de.svg
Davide Berasi
daviBera
user
daviBera
69a958ccddca602df39671f2
Alessandro Conti
claimed_verified
2026-03-06T13:56:35.656Z
633f243c13e836a0fc507388
https://cdn-avatars.huggingface.co/v1/production/uploads/633f243c13e836a0fc507388/Td8TWn1q2L78sZN9098AO.jpeg
Alessandro Conti
altndrr
user
altndrr
69a958ccddca602df39671f3
Elisa Ricci
69a958ccddca602df39671f4
Yiming Wang
Specificity-aware reinforcement learning for fine-grained open-world classification
Classifying fine-grained visual concepts under open-world settings, i.e., without a predefined label set, demands models to be both accurate and specific. Recent reasoning Large Multimodal Models (LMMs) exhibit strong visual understanding capability but tend to produce overly generic predictions when performing fine-grained image classification. Our preliminary analysis reveals that models do possess the intrinsic fine-grained domain knowledge. However, promoting more specific predictions (specificity) without compromising correct ones (correctness) remains a non-trivial and understudied challenge. In this work, we investigate how to steer reasoning LMMs toward predictions that are both correct and specific. We propose a novel specificity-aware reinforcement learning framework, SpeciaRL, to fine-tune reasoning LMMs on fine-grained image classification under the open-world setting. SpeciaRL introduces a dynamic, verifier-based reward signal anchored to the best predictions within online rollouts, promoting specificity while respecting the model's capabilities to prevent incorrect predictions. Our out-of-domain experiments show that SpeciaRL delivers the best trade-off between correctness and specificity across extensive fine-grained benchmarks, surpassing existing methods and advancing open-world fine-grained image classification. Code and model are publicly available at https://github.com/s-angheben/SpeciaRL.
A novel reinforcement learning framework called SpeciaRL is proposed to improve the specificity of large multimodal models in open-world fine-grained image classification while maintaining prediction correctness.
Large Multimodal Models reinforcement learning specificity open-world setting fine-grained image classification verifier-based reward signal online rollouts