On-Policy Self-Distillation with Sampled Demonstrations Reduces Output Diversity
Andrei Liviu Nicolicioiu, Mohammad Pezeshki, Aaron Courville
Read on arXiv →
Key claim
Self-distillation can reduce output diversity despite high accuracy.
In plain English
Imagine you're trying to build a model that learns from its own past performances, using its own outputs as a guide. This is a bit like a student learning from their own previous tests, but it can lead to problems. When a model uses its own past successes to teach itself, it might become overly confident in certain strategies, ignoring other potentially useful approaches. This is what happens in self-distillation, where the model acts as both teacher and student. The downside is that while it can achieve high accuracy, it often loses the ability to generate diverse solutions, especially when faced with new or unexpected challenges. This paper dives into that issue, showing that the model's feedback can amplify its existing biases, leading to a lack of variety in its responses. The authors provide a theoretical framework to understand this behavior and demonstrate it through experiments on specific tasks. The practical takeaway is that while self-distillation can be effective, it may not be the best choice when you need a model that can adapt to a wide range of situations.
The paper introduces a new approach to self-distillation that highlights the trade-offs in diversity and accuracy.
The claims are supported by theoretical analysis and empirical results across multiple tasks.
Deep reliability assessment
The methodology supports the claim that self-distillation with sampled correct demonstrations can reduce functional and semantic rollout diversity on the tested graph path-finding and ScienceQA settings, even when pass@1 is competitive. The broader claim about self-distillation in general is less supported, because the analysis excludes richer teacher contexts like runtime errors, environmental feedback, external verifiers, and assumes a frozen/base-policy teacher rather than the EMA/self-generated teachers often used in practice.
Reproducibility
Partial: the paper reports model families, training setup, seeds, GPU budget, datasets such as SciKnowEval, and references the open-source nano-aha-moment RL library, but the excerpt does not mention a dedicated repository for this paper's exact experiments or graph-task data generation scripts.
Key figure
Figure 1 contrasts GRPO, which gives equal reward to all correct rollouts, with SDSD, where a teacher conditioned on one correct demonstration reinforces similar rollouts more strongly, leading to flatter pass@k curves and lower functional diversity.