Why Multi-Step Tool-Use Reinforcement Learning Collapses and How Supervisory Signals Fix It
Yupu Hao, Zhuoran Jin, Huanxuan Liao, Kang Liu, Jun Zhao
Read on arXiv →
Key claim
Interleaving supervised fine-tuning with RL improves model stability.
In plain English
Imagine you're trying to build a system that can use tools to perform complex tasks, like scheduling meetings or fetching information. You might think that just training a model with reinforcement learning (RL) would be enough, but it often leads to problems. Sometimes, the model's performance can suddenly drop, a situation known as catastrophic collapse, where it fails to use tools correctly. This happens because the model gets confused by unexpected spikes in certain control signals, which disrupt its ability to execute tasks properly. The underlying capability to use tools is still there, but it gets hidden by the way the model is trained or the formats it encounters.
To tackle these issues, the authors explore various supervisory signals, like off-policy supervision and hint-based guidance, applied in different training setups. They find that mixing supervised fine-tuning with RL helps stabilize the model's performance, making it more reliable in using tools. However, they also discover that when the model faces data that is different from what it was trained on, its performance can suffer. This research emphasizes the need to understand the failures of RL in tool use and shows how diverse training signals can help guide the learning process, ultimately leading to more robust models for complex tasks.
The paper introduces a systematic investigation of diverse supervisory signals to improve stability in tool-use tasks, extending existing reinforcement learning methods.
The findings are supported by experiments that analyze various training schemes and their effects on model performance.
Deep reliability assessment
The methodology supports the claim that, in their limited open-source verifiable tool-use setup with Qwen2.5-1.5B-Instruct and Qwen3-1.7B, pure RL can suffer structural collapse and interleaved SFT-style supervision can stabilize training. It is less conclusive on whether the identified control-token probability spikes are the dominant failure mode across larger models, different tool schemas, production agent frameworks, or larger-scale environments, especially since data scale is explicitly not explored.
Reproducibility
Code is open sourced at the GitHub URL stated in the abstract. The paper appears to use open-source/verifiable tool-invoking environments and datasets such as ToolACE/BFCL-style supervision, but the provided excerpt does not confirm that all training/evaluation data and exact generated trajectories are released.
Key figure
Figure 1 tracks token/output category frequencies during Qwen2.5-1.5B-Instruct RL training, showing a transition from healthy tool calls and responses toward polluted outputs and eventual structural collapse.