In-Context Reinforcement Learning for Tool Use in Large Language Models
While large language models (LLMs) exhibit strong reasoning abilities, their performance on complex tasks is often constrained by the limitations of their internal knowledge. A compelling approach to overcome this challenge is to augment these models with external tools -- such as Python interpreters for mathematical computations or search engines for retrieving factual information. However, enabling models to use these tools effectively remains a significant challenge. Existing methods typically rely on cold-start pipelines that begin with supervised fine-tuning (SFT), followed by reinforcement learning (RL). These approaches often require substantial amounts of labeled data for SFT, which is expensive to annotate or synthesize. In this work, we propose In-Context Reinforcement Learning (ICRL), an RL-only framework that eliminates the need for SFT by leveraging few-shot prompting during the rollout stage of RL. Specifically, ICRL introduces in-context examples within the rollout prompts to teach the model how to invoke external tools. Furthermore, as training progresses, the number of in-context examples is gradually reduced, eventually reaching a zero-shot setting where the model learns to call tools independently. We conduct extensive experiments across a range of reasoning and tool-use benchmarks. Results show that ICRL achieves state-of-the-art performance, demonstrating its effectiveness as a scalable, data-efficient alternative to traditional SFT-based pipelines.
69afaf0cb3ab31dee1281eb2
Yaoqi Ye
claimed_verified
2026-03-13T08:14:46.127Z
6824f899e8acf803b47d9d44
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/bt8QoTHOZ0m-SboYIQeVd.png
Ye Yaoqi
shadowapplese
user
shadowapplese
69afaf0cb3ab31dee1281eb3
Yiran Zhao
claimed_verified
2026-03-12T09:22:17.138Z
64802face9ff472e30dc1ceb
https://cdn-avatars.huggingface.co/v1/production/uploads/64802face9ff472e30dc1ceb/bcwTlgpaUrU7m2RMB5zCc.png
Yiran Zhao
Yiran0924
user
Yiran0924
69afaf0cb3ab31dee1281eb4
Keyu Duan
admin_assigned
2026-03-12T09:32:13.373Z
6336a11331efcb5647ef32c8
https://cdn-avatars.huggingface.co/v1/production/uploads/6336a11331efcb5647ef32c8/mM0eC0mIV4W4W7-R-cleV.png
Keyu Duan
vermouthdky
user
vermouthdky
69afaf0cb3ab31dee1281eb5
Zeyu Zheng
admin_assigned
2026-03-12T09:32:34.631Z
67cffe31d6f5c181e523610c
/avatars/8706e09fc4397456baeddb8942c8b0ee.svg
Zeyu Zheng
zeyu-zheng
user
zeyu-zheng
69afaf0cb3ab31dee1281eb6
Kenji Kawaguchi
69afaf0cb3ab31dee1281eb7
Cihang Xie
admin_assigned
2026-03-12T09:32:43.710Z
645eb61da3c5cd8a16efffff
/avatars/9112bfeed598dfabf9e077e69e09ecc9.svg
Cihang Xie
cihangxie
user
cihangxie
69afaf0cb3ab31dee1281eb8
Michael Qizhe Shieh
In-Context Reinforcement Learning for Tool Use in Large Language Models
While large language models (LLMs) exhibit strong reasoning abilities, their performance on complex tasks is often constrained by the limitations of their internal knowledge. A compelling approach to overcome this challenge is to augment these models with external tools -- such as Python interpreters for mathematical computations or search engines for retrieving factual information. However, enabling models to use these tools effectively remains a significant challenge. Existing methods typically rely on cold-start pipelines that begin with supervised fine-tuning (SFT), followed by reinforcement learning (RL). These approaches often require substantial amounts of labeled data for SFT, which is expensive to annotate or synthesize. In this work, we propose In-Context Reinforcement Learning (ICRL), an RL-only framework that eliminates the need for SFT by leveraging few-shot prompting during the rollout stage of RL. Specifically, ICRL introduces in-context examples within the rollout prompts to teach the model how to invoke external tools. Furthermore, as training progresses, the number of in-context examples is gradually reduced, eventually reaching a zero-shot setting where the model learns to call tools independently. We conduct extensive experiments across a range of reasoning and tool-use benchmarks. Results show that ICRL achieves state-of-the-art performance, demonstrating its effectiveness as a scalable, data-efficient alternative to traditional SFT-based pipelines.
In-Context Reinforcement Learning (ICRL) enables large language models to effectively use external tools through a reinforcement learning-only framework that eliminates the need for costly supervised fine-tuning by gradually reducing in-context examples during training.
large language models reinforcement learning supervised fine-tuning tool usage in-context learning few-shot prompting rollout stage external tools mathematical computations search engines factual information retrieval state-of-the-art performance data efficiency