From Raw Experience to Skill Consumption: A Systematic Study of Model-Generated Agent Skills
Zisu Huang, Jingwen Xu, Yifan Yang, Ziyang Gong, Qihao Yang, Muzhao Tian, Xiaohua Wang, Changze Lv, Xuemei Gao, Qi Dai, Bei Liu, Kai Qiu, Xue Yang, Dongdong Chen, Xiaoqing Zheng, Chong Luo
Key claim
Model-generated skills improve performance but may cause negative transfer.
This paper investigates the lifecycle of skills in language agents, focusing on their extraction and consumption. A key finding is that model-generated skills generally improve performance but can lead to negative transfer, highlighting the complexity of skill utility across different models. The authors propose a meta-skill to enhance skill extraction and reduce negative transfer.
The paper introduces a comprehensive evaluation framework for skill lifecycle in language agents.
The study includes systematic experimental results across multiple domains, though it lacks some methodological details.
Deep reliability assessment
The methodology supports a systematic evaluation of model-generated skills across diverse domains, but the results are contingent on the specific models and domains tested, which may not generalize universally.
Reproducibility
Yes, the study provides open-source code available at https://aka.ms/SkillLens, which allows for reproducibility of the experiments and further exploration.
Discussion questions
- How does the success-failure composition of experience pools affect skill extraction across different domains, and could this be generalized?
- What are the practical implications of negative transfer in real-world applications, and how can developers mitigate these risks?
- What specific conditions or scenarios would demonstrate that model-generated skills do not consistently improve agent performance across different domains?
Key figure
Figure 1 provides an overview of the study design, illustrating the full trajectory-to-skill lifecycle across experience generation, skill extraction, and skill consumption stages.