ASTRA: Automated Synthesis of agentic Trajectories and Reinforcement Arenas
Large language models (LLMs) are increasingly used as tool-augmented agents for multi-step decision making, yet training robust tool-using agents remains challenging. Existing methods still require manual intervention, depend on non-verifiable simulated environments, rely exclusively on either supervised fine-tuning (SFT) or reinforcement learning (RL), and struggle with stable long-horizon, multi-turn learning. To address these challenges, we introduce ASTRA, a fully automated end-to-end framework for training tool-augmented language model agents via scalable data synthesis and verifiable reinforcement learning. ASTRA integrates two complementary components. First, a pipeline that leverages the static topology of tool-call graphs synthesizes diverse, structurally grounded trajectories, instilling broad and transferable tool-use competence. Second, an environment synthesis framework that captures the rich, compositional topology of human semantic reasoning converts decomposed question-answer traces into independent, code-executable, and rule-verifiable environments, enabling deterministic multi-turn RL. Based on this method, we develop a unified training methodology that integrates SFT with online RL using trajectory-level rewards to balance task completion and interaction efficiency. Experiments on multiple agentic tool-use benchmarks demonstrate that ASTRA-trained models achieve state-of-the-art performance at comparable scales, approaching closed-source systems while preserving core reasoning ability. We release the full pipelines, environments, and trained models at https://github.com/LianjiaTech/astra.
697c279ea67238fac88cc104
Xiaoyu Tian
claimed_verified
2026-02-02T17:00:01.968Z
621499d72be42a56cca7afad
/avatars/3cea70d1a55c8096b4270093c69e4a5e.svg
TianXiaoyu
Emperorizzis
user
Emperorizzis
697c279ea67238fac88cc105
Haotian Wang
697c279ea67238fac88cc106
Shuaiting Chen
697c279ea67238fac88cc107
Hao Zhou
697c279ea67238fac88cc108
Kaichi Yu
697c279ea67238fac88cc109
Yudian Zhang
697c279ea67238fac88cc10a
Jade Ouyang
claimed_verified
2026-02-02T16:59:59.965Z
690189db2b5a1d242306b77f
/avatars/b0297c57395029da2725758671952bb6.svg
jade ouyang
jade0101
user
jade0101
697c279ea67238fac88cc10b
Junxi Yin
697c279ea67238fac88cc10c
Jiong Chen
697c279ea67238fac88cc10d
Baoyan Guo
697c279ea67238fac88cc10e
Lei Zhang
697c279ea67238fac88cc10f
Junjie Tao
697c279ea67238fac88cc110
Yuansheng Song
697c279ea67238fac88cc111
Ming Cui
697c279ea67238fac88cc112
Chengwei Liu
claimed_verified
2026-02-09T08:36:34.981Z
6417d7e01f1f3b0fa8112f8a
/avatars/fe1ae77306f4a8fe1721e6eca7b5cc2d.svg
ChengweiLiu
lcw888
user
lcw888
ASTRA: Automated Synthesis of agentic Trajectories and Reinforcement Arenas
Large language models (LLMs) are increasingly used as tool-augmented agents for multi-step decision making, yet training robust tool-using agents remains challenging. Existing methods still require manual intervention, depend on non-verifiable simulated environments, rely exclusively on either supervised fine-tuning (SFT) or reinforcement learning (RL), and struggle with stable long-horizon, multi-turn learning. To address these challenges, we introduce ASTRA, a fully automated end-to-end framework for training tool-augmented language model agents via scalable data synthesis and verifiable reinforcement learning. ASTRA integrates two complementary components. First, a pipeline that leverages the static topology of tool-call graphs synthesizes diverse, structurally grounded trajectories, instilling broad and transferable tool-use competence. Second, an environment synthesis framework that captures the rich, compositional topology of human semantic reasoning converts decomposed question-answer traces into independent, code-executable, and rule-verifiable environments, enabling deterministic multi-turn RL. Based on this method, we develop a unified training methodology that integrates SFT with online RL using trajectory-level rewards to balance task completion and interaction efficiency. Experiments on multiple agentic tool-use benchmarks demonstrate that ASTRA-trained models achieve state-of-the-art performance at comparable scales, approaching closed-source systems while preserving core reasoning ability. We release the full pipelines, environments, and trained models at https://github.com/LianjiaTech/astra.
ASTRA is an automated framework that trains tool-augmented language models using synthetic data and verifiable reinforcement learning to improve multi-step decision-making capabilities.
tool-call graphs trajectory-level rewards supervised fine-tuning reinforcement learning agent training multi-step decision making verifiable environments compositional topology semantic reasoning tool-augmented language models