Weak-to-Strong Generalization via Direct On-Policy Distillation

Imagine you're trying to improve a powerful language model, but every time you want to enhance it, you have to run expensive reinforcement learning (RL) training, which involves generating a lot of data. This can be a huge bottleneck, especially as models get larger. The traditional approach is to train the strong model directly, but this can be inefficient and slow, especially when the model is already complex. This is what's called the post-training bottleneck, where the cost of training outweighs the benefits of the improvements you might get.

What this paper proposes is a clever workaround: instead of training the strong model directly, you first train a smaller, cheaper model using RL. Then, you take what that smaller model learned and apply it to the stronger model. However, simply copying the smaller model's learned behavior isn't enough because it might carry over its own limitations. Instead, the authors introduce a method called Direct On-Policy Distillation (Direct-OPD), which focuses on transferring the specific improvements the smaller model made during its training. This way, the stronger model can learn from the weaker one without inheriting its flaws.

The results are promising. By using this method, they were able to significantly improve the performance of a model called Qwen3-1.7B in a short amount of time. This means that instead of always needing to retrain the strong model from scratch, you can leverage the learning from a weaker model, making the process faster and more efficient. For anyone building language models, this approach could save a lot of time and resources while still achieving better results.

The approach introduces a new method for leveraging weaker models to enhance stronger ones, which is a significant extension of existing reinforcement learning techniques.

The empirical results show consistent improvements across models, though the evaluation could benefit from more diverse baselines.