DemoPSD: Disagreement-Modulated Policy Self-Distillation
Yunhe Li, Hao Shi, Wenhao Liu, Mengzhe Ruan, Hanxu Hou, Zhongxiang Dai, Shuang Qiu, Linqi Song
Read on arXiv →
Key claim
DemoPSD reduces information leakage and enhances exploration.
In plain English
Imagine you're training a language model to answer questions across various topics. You want it to learn effectively from its own experiences while also benefiting from guidance provided by a teacher model. However, if the teacher model gives too much specific information, the student might overfit to those details and struggle when faced with new, unseen questions. This is a common issue known as privileged information leakage, where the student learns shortcuts that don't apply in real-world scenarios. Additionally, the student might lose its ability to explore different reasoning paths, which is crucial for tackling diverse problems.
Currently, many approaches rely on dense supervision from the teacher, which can lead to these pitfalls. The paper introduces a new method called DemoPSD, which allows the student to learn from the teacher without fully relying on its guidance. Instead of just mimicking the teacher's outputs, DemoPSD encourages the student to maintain its own reasoning abilities while selectively adopting useful insights from the teacher. This is achieved through a clever mechanism that balances the learning from both models, ensuring that the student doesn't just memorize answers but also develops a deeper understanding.
The results show that DemoPSD not only mitigates the leakage of privileged information but also preserves the student's capacity to explore different solutions. In practical terms, this means that when you're deploying language models in real-world applications, you can expect them to perform better on new types of questions, thanks to the improved training approach. This is particularly important for applications that require robust generalization across various domains.
The approach introduces a new framework for balancing teacher and student learning in self-distillation, addressing significant issues in existing methods.
The claims are supported by extensive experiments across multiple domains, demonstrating solid performance improvements.
Deep reliability assessment
The methodology supports the claim that DemoPSD can mitigate privileged information leakage and preserve exploration capacity, but the generalization claims may be overextended without testing under different types of distribution shifts like label shift.
Reproducibility
No open source code or dataset URL is mentioned in the paper.
Key figure
Figure 1 illustrates that DemoPSD maintains higher policy entropy over training steps compared to SDPO, which translates into better best@16 performance across domains.