TestEvo-Bench: An Executable and Live Benchmark for Test and Code Co-Evolution
Jiale Amber Wang, Kaiyuan Wang, Pengyu Nie
Read on arXiv →
Key claim
TestEvo-Bench improves evaluation of test automation agents.
In plain English
Imagine you're working on a software project where every time you change the code, you need to ensure that the tests reflect those changes. This is crucial because outdated tests can lead to bugs slipping through. Currently, many test generation methods don't consider the actual code changes, which can result in tests that are either irrelevant or fail to run properly. This disconnect is problematic because it makes it hard to know if the tests are truly capturing the new behavior of the software. This issue is known as the lack of semantic alignment between tests and code changes.
To address this, the authors created TestEvo-Bench, a benchmark that focuses on the co-evolution of tests and code. It includes tasks where agents must either generate new tests or update existing ones based on real code changes from software repositories. Each task is tied to actual commit histories, ensuring that the tests are relevant and executable. The benchmark also tracks the timing of changes to prevent data leakage during evaluation. With 746 test generation and 509 test update tasks from a large dataset, the authors tested several advanced agents and found that they could achieve a success rate of up to 77.5% for generating new tests. However, they noted that performance declines on the most recent tasks, indicating that adapting to ongoing changes in code is still a challenge.
This approach is a step forward from previous methods that often isolated tests from the code changes, providing a more realistic evaluation of how well automation agents can keep up with evolving software. For builders, this means that using TestEvo-Bench can help ensure that the tests you rely on are not just theoretically sound but practically effective in real-world scenarios.
The introduction of a benchmark that integrates test generation and code evolution is a significant advancement in evaluating test automation.
The paper provides solid experimental results with a well-defined benchmark, though it could benefit from more extensive baselines.
Deep reliability assessment
The methodology supports evaluating test and code co-evolution by using real commit histories and execution-grounded metrics, but it may overclaim by not fully addressing the challenges of recent benchmark tasks and cost limitations.
Reproducibility
Yes, the TESTEVO-BENCH leaderboard and data explorer are hosted at https://www.testevo-bench.com.
Key figure
Figure 1 likely illustrates the architecture or workflow of the TESTEVO-BENCH benchmark, showing how test and code co-evolution tasks are mined and evaluated.