SWE-INTERACT: Reimagining SWE Benchmarks as User-Driven Long-Horizon Coding Sessions

Imagine you're building a coding assistant that helps developers write software. In a typical setup, you might give the assistant a complete set of requirements and let it work autonomously. However, this doesn't reflect how real developers operate, where they often start with vague ideas and refine them through conversation and feedback. The problem is that when coding agents are tested in these straightforward scenarios, they perform well, but they often fail when faced with the ambiguity and evolving requirements of real-world tasks. This is what's called a lack of adaptability in multi-turn interactions.

To address this, SWE-Interact creates a more realistic testing environment where coding agents interact with a user simulator that gradually reveals requirements and provides feedback. This setup allows the agents to demonstrate their ability to understand user intent and adapt to changing instructions. The key insight is that coding agents need to be evaluated not just on their ability to implement code but also on how well they can navigate a dynamic conversation with a user.

The results show that even the best models, like Opus 4.8 and GPT 5.5, struggle in this interactive setting, solving only 25% of tasks compared to 50% in simpler scenarios. This indicates that while these models can handle initial ambiguity, they still face challenges like forgetting requirements and making technical mistakes. For builders, this means that if you're developing coding agents, you need to focus on enhancing their interactive capabilities to better support real-world software development workflows.

SWE-Interact introduces a new evaluation framework that reflects real-world software engineering tasks.

The findings are supported by large-scale studies and comparisons with existing benchmarks.