EVA: Efficient Reinforcement Learning for End-to-End Video Agent
Video understanding with multimodal large language models (MLLMs) remains challenging due to the long token sequences of videos, which contain extensive temporal dependencies and redundant frames. Existing approaches typically treat MLLMs as passive recognizers, processing entire videos or uniformly sampled frames without adaptive reasoning. Recent agent-based methods introduce external tools, yet still depend on manually designed workflows and perception-first strategies, resulting in inefficiency on long videos. We present EVA, an Efficient Reinforcement Learning framework for End-to-End Video Agent, which enables planning-before-perception through iterative summary-plan-action-reflection reasoning. EVA autonomously decides what to watch, when to watch, and how to watch, achieving query-driven and efficient video understanding. To train such agents, we design a simple yet effective three-stage learning pipeline - comprising supervised fine-tuning (SFT), Kahneman-Tversky Optimization (KTO), and Generalized Reward Policy Optimization (GRPO) - that bridges supervised imitation and reinforcement learning. We further construct high-quality datasets for each stage, supporting stable and reproducible training. We evaluate EVA on six video understanding benchmarks, demonstrating its comprehensive capabilities. Compared with existing baselines, EVA achieves a substantial improvement of 6-12% over general MLLM baselines and a further 1-3% gain over prior adaptive agent methods. Our code and model are available at https://github.com/wangruohui/EfficientVideoAgent.
69c3924c95b48f16a300792b
Yaolun Zhang
claimed_verified
2026-03-26T13:43:33.405Z
648d2e2e514bf0ce32ba729f
https://cdn-avatars.huggingface.co/v1/production/uploads/648d2e2e514bf0ce32ba729f/VPL1rehLxkvixz5oRD6u_.jpeg
Yaolun Zhang
Mercury7353
user
Mercury7353
69c3924c95b48f16a300792c
Ruohui Wang
69c3924c95b48f16a300792d
Jiahao Wang
claimed_verified
2026-03-26T13:43:35.232Z
66bc7862aa7cdcb1c31a1efb
/avatars/4c2ab907247fe071ff5cdd71c404ca7c.svg
wang jiahao
datamonkey
user
datamonkey
69c3924c95b48f16a300792e
Yepeng Tang
69c3924c95b48f16a300792f
Xuanyu Zheng
69c3924c95b48f16a3007930
Haonan Duan
69c3924c95b48f16a3007931
Hao Lu
69c3924c95b48f16a3007932
Hanming Deng
69c3924c95b48f16a3007933
Lewei Lu
EVA: Efficient Reinforcement Learning for End-to-End Video Agent
Video understanding with multimodal large language models (MLLMs) remains challenging due to the long token sequences of videos, which contain extensive temporal dependencies and redundant frames. Existing approaches typically treat MLLMs as passive recognizers, processing entire videos or uniformly sampled frames without adaptive reasoning. Recent agent-based methods introduce external tools, yet still depend on manually designed workflows and perception-first strategies, resulting in inefficiency on long videos. We present EVA, an Efficient Reinforcement Learning framework for End-to-End Video Agent, which enables planning-before-perception through iterative summary-plan-action-reflection reasoning. EVA autonomously decides what to watch, when to watch, and how to watch, achieving query-driven and efficient video understanding. To train such agents, we design a simple yet effective three-stage learning pipeline - comprising supervised fine-tuning (SFT), Kahneman-Tversky Optimization (KTO), and Generalized Reward Policy Optimization (GRPO) - that bridges supervised imitation and reinforcement learning. We further construct high-quality datasets for each stage, supporting stable and reproducible training. We evaluate EVA on six video understanding benchmarks, demonstrating its comprehensive capabilities. Compared with existing baselines, EVA achieves a substantial improvement of 6-12% over general MLLM baselines and a further 1-3% gain over prior adaptive agent methods. Our code and model are available at https://github.com/wangruohui/EfficientVideoAgent.
EVA is an efficient reinforcement learning framework for video understanding that enables adaptive reasoning through iterative planning and attention mechanisms, outperforming existing methods on multiple video benchmarks.
multimodal large language models reinforcement learning video understanding iterative summary-plan-action-reflection supervised fine-tuning Kahneman-Tversky Optimization Generalized Reward Policy Optimization adaptive reasoning attention mechanisms