Surrogate Fidelity: When Can Open LLMs Explain Closed Ones?
Philippe Chlenski, Zachariah Carmichael, Ayush Warikoo, Chia-Tse Shao, Yingxiao Ye, Aobo Yang, Vivek Miglani, Nehal Bandi
Read on arXiv →
Key claim
Prediction agreement does not ensure causal understanding.
In plain English
Imagine you're trying to understand how a complex AI model makes decisions, but you only have limited access to its inner workings. You might be able to see what it predicts, but not why it makes those predictions. This is a common issue in machine learning, especially with popular language models that only expose certain outputs, like log-probabilities. When you try to interpret these models, you might think you understand them based on their predictions, but that can be misleading. This is known as the surrogate problem: just because two models predict the same outcome doesn't mean they operate in the same way or for the same reasons.
In this paper, the authors explore how to evaluate the fidelity of these surrogate models at different levels, such as prediction and attribution. They find that while models may agree on what the answer is, they often disagree on why they arrived at that answer. This leads to a significant insight: the signals we can observe from models, like attention patterns, might be stable across different models, but they don't necessarily help us understand the causal reasons behind their decisions. This mismatch is crucial because it means that insights gained from one model may not apply to another, especially if the latter is a closed model.
What this means for builders is that when you're working with AI models, especially in applications where understanding the reasoning is important, you need to be cautious about assuming that insights from one model will transfer to another. The findings suggest that just because two models predict the same thing, it doesn't mean they share the same underlying logic. This paper provides a framework for thinking about these issues, which can help in developing more reliable and interpretable AI systems.
The paper introduces a new perspective on surrogate fidelity in mechanistic interpretability, highlighting the limitations of current methods.
The findings are supported by evaluations across multiple models and tasks, providing a solid basis for the claims made.
Deep reliability assessment
The methodology supports the claim that prediction fidelity often overstates attribution fidelity, highlighting the complexity of transferring mechanistic insights from open to closed models. However, the paper may overclaim the generalizability of these findings across all model types and tasks without further empirical validation.
Reproducibility
yes, the paper mentions that code and results are available at https://github.com/facebookresearch/surrogate.
Key figure
Figure 1 illustrates the surrogate fidelity evaluation pipeline, showing how prediction log-odds, ablation-based attributions, and perturbation responses are extracted from models to assess prediction, attribution, and representation fidelity.