When Does Combining Language Models Help? A Co-Failure Ceiling on Routing, Voting, and Mixture-of-Agents Across 67 Frontier Models
Josef Chen
Read on arXiv →
Key claim
Multi-model accuracy is limited by shared error rates.
In plain English
Imagine you're trying to build a system that uses multiple models to answer questions, like a team of experts. The idea is that by having different models tackle different types of questions, you can get better overall accuracy. However, there's a catch: if multiple models are wrong on the same question, your accuracy can't improve beyond a certain point, which is determined by how often they fail together. This is known as the beta rate, and it’s a crucial factor that many people don’t consider when designing these systems.
Currently, when people combine models, they often rely on metrics like average pairwise error correlation to gauge performance. But this can be misleading because two models can have the same correlation yet perform very differently in terms of their collective failure rates. This is where the paper steps in. It provides a new way to measure this beta rate and shows that understanding it can help you make better decisions about which models to combine and how to route queries.
The key takeaway is that simply adding more models doesn't guarantee better performance. In fact, the research found that in many cases, the best single model outperformed combinations of models unless there was a strong signal guiding which model to use for each question. This insight is particularly valuable for anyone building systems that rely on multiple models, as it emphasizes the importance of understanding model interactions and error rates.
The paper introduces a new perspective on the limitations of multi-model systems by quantifying the error correlation in a novel way.
The findings are supported by extensive empirical analysis across multiple models and tasks.
Deep reliability assessment
The methodology strongly supports the formal ceiling for policies that must output one member model's answer, and the empirical claim that β can be measured on answer-checkable held-out sets. The broader 'market-scale' conclusion is less secure because some decisive all-models-wrong counts are small, the learned routers tested may not represent the best possible production router, and results may not transfer to tasks where correctness is subjective or where fusion can synthesize a new answer.
Reproducibility
No public code repository is mentioned. The paper uses standard benchmarks and says model snapshots, prices, rosters, costs, and a registry are frozen in appendices, but from the provided text there is no open-source code or released dataset URL.
Key figure
Figure 1 shows a query entering an orchestration layer that routes, cascades, or fuses across a priced pool of 67 correlated models, with overall accuracy capped by 1 − β and two regimes highlighted: ceiling-bound open-ended math and realizability-bound science.