Agon: Competitive Cross-Model RL with Implicit Rival Grading of Reasoning

Imagine you're trying to build a model that not only gives answers but also thinks critically about how it arrives at those answers. Traditional reinforcement learning methods often focus on the final output, which can lead to models that generate more text without necessarily improving their reasoning skills. This is a problem because it means the model might not be learning to think better, just to produce more content. This issue is known as the lack of graded reasoning, where the process of arriving at an answer isn't evaluated, leaving a gap in the model's learning process.

To address this, the paper introduces a new approach where two models compete against each other. Each model takes turns drafting a solution to a problem while the other reads and critiques it. This setup allows them to implicitly evaluate each other's reasoning without needing explicit labels for good thinking. The idea is that by trying to out-reason each other, both models become progressively better, as they face increasingly strong opponents. This competitive dynamic is a shift from traditional single-model reinforcement learning, which often lacks this level of interaction.

In practical terms, this method has shown to significantly improve performance on difficult tasks, such as those found in the DeepMath benchmark, where it doubled the pass rate compared to existing methods. For anyone building reasoning models, this competitive framework offers a promising way to enhance their capabilities without the need for detailed grading systems.

The approach introduces a novel competitive grading mechanism between models that enhances reasoning capabilities.

The results are supported by strong performance improvements on specific benchmarks, though broader validation could strengthen claims.