VidEmo: Affective-Tree Reasoning for Emotion-Centric Video Foundation
  Models
Understanding and predicting emotion from videos has gathered significant
attention in recent studies, driven by advancements in video large language
models (VideoLLMs). While advanced methods have made progress in video emotion
analysis, the intrinsic nature of emotions poses significant challenges.
Emotions are characterized by dynamic and cues-dependent properties, making it
difficult to understand complex and evolving emotional states with reasonable
rationale. To tackle these challenges, we propose a novel affective cues-guided
reasoning framework that unifies fundamental attribute perception, expression
analysis, and high-level emotional understanding in a stage-wise manner. At the
core of our approach is a family of video emotion foundation models (VidEmo),
specifically designed for emotion reasoning and instruction-following. These
models undergo a two-stage tuning process: first, curriculum emotion learning
for injecting emotion knowledge, followed by affective-tree reinforcement
learning for emotion reasoning. Moreover, we establish a foundational data
infrastructure and introduce a emotion-centric fine-grained dataset (Emo-CFG)
consisting of 2.1M diverse instruction-based samples. Emo-CFG includes
explainable emotional question-answering, fine-grained captions, and associated
rationales, providing essential resources for advancing emotion understanding
tasks. Experimental results demonstrate that our approach achieves competitive
performance, setting a new milestone across 15 face perception tasks.
690abebbd70e173c84528fd5
Zhicheng Zhang
690abebbd70e173c84528fd6
Weicheng Wang
690abebbd70e173c84528fd7
Yongjie Zhu
690abebbd70e173c84528fd8
Wenyu Qin
690abebbd70e173c84528fd9
Pengfei Wan
690abebbd70e173c84528fda
Di Zhang
690abebbd70e173c84528fdb
Jufeng Yang
VidEmo: Affective-Tree Reasoning for Emotion-Centric Video Foundation
  Models
Understanding and predicting emotion from videos has gathered significant
attention in recent studies, driven by advancements in video large language
models (VideoLLMs). While advanced methods have made progress in video emotion
analysis, the intrinsic nature of emotions poses significant challenges.
Emotions are characterized by dynamic and cues-dependent properties, making it
difficult to understand complex and evolving emotional states with reasonable
rationale. To tackle these challenges, we propose a novel affective cues-guided
reasoning framework that unifies fundamental attribute perception, expression
analysis, and high-level emotional understanding in a stage-wise manner. At the
core of our approach is a family of video emotion foundation models (VidEmo),
specifically designed for emotion reasoning and instruction-following. These
models undergo a two-stage tuning process: first, curriculum emotion learning
for injecting emotion knowledge, followed by affective-tree reinforcement
learning for emotion reasoning. Moreover, we establish a foundational data
infrastructure and introduce a emotion-centric fine-grained dataset (Emo-CFG)
consisting of 2.1M diverse instruction-based samples. Emo-CFG includes
explainable emotional question-answering, fine-grained captions, and associated
rationales, providing essential resources for advancing emotion understanding
tasks. Experimental results demonstrate that our approach achieves competitive
performance, setting a new milestone across 15 face perception tasks.
A novel affective cues-guided reasoning framework using video emotion foundation models and a fine-grained dataset achieves competitive performance in emotion understanding tasks.
video large language models VideoLLMs video emotion analysis affective cues-guided reasoning video emotion foundation models VidEmo curriculum emotion learning affective-tree reinforcement learning emotion-centric fine-grained dataset Emo-CFG explainable emotional question-answering fine-grained captions face perception tasks