Has Anyone Tried Running Only PR-Relevant Tests?

[younggee123456]

Hi! We've found that a significant portion of the repository construction process is spent running the entire test suite and parsing the test logs.

We're wondering whether it would be possible to first identify a subset of test cases that are relevant to a given PR, and then only validate those test cases afterward.

Have you ever tried something similar? If so, we'd really appreciate it if you could share your experience or any insights.

[njukenanli]

Thank your for providing this valuable observation. I will provide two suggestions here.

Modify the RepoLaunch workflow

Our original reason to build and test the WHOLE repository is because if we only build or test part of a repository, we cannot verify whether a repo build is successful or failed/partial. Therefore, RepoLaunch is intentionally designed to build and test the whole repository.

You can of course customize RepoLaunch with your own fork like this:

1. Extract PR-relevant test files from the test patch.
2. Directly modify the launch/agent/setup/verify.py at the first setup stage to prompt the agent to get ONLY the test command that specifies the PR-related test files.
3. Then at launch/agent/organzie/testall.py at the second organize stage, also prompt agent to generate the test command to specify the PR-related test files.

Extract PR-specific test command from RepoLaunch result

If you don't want to modify RepoLaunch source code and extracting PR-specific test command from RepoLaunch result is acceptable, here are some suggestions.

For the old manually-crafted SWE-bench dataset, each task instance only runs the test files where F2P test cases lie in to verify. The relevant test files are manually selected. However, for automated pipeline this is not trivial. For easy languages like Python, one can easily specify keyword in test name to run like pytest -k 'keyword', which is the Python SWE benchmarks usually do. But this pattern does not generalize to other languages.

RepoLaunch designs an additional step at the end of the organize stage to map each test case name to the per-test command that specify that test case to run (See docs/Development.md). I think this can help you select PR-related test files to run like this:

Extract names of relevant tests from test patch or F2P test list.
-> map the selected test names to the corresponding per-test command generated by RepoLaunch at `launch/agent/organize/testone.py`
-> merge the per-test commands by test file path prefix if possible, so that Pass_to_Pass tests are included by the merged test command

But this process is language-specific, different languages need different extract/merge rules. And old languages like C/C++ usually don't support specifying tests to run.

Still, this is an open question. I will leave the issue open for community discussion.

[younggee123456]

Thank you very much for your prompt reply! I was wondering if you've explored any other approaches or have any recommendations for accelerating the repository construction process. For example, the paper mentions using a hard-coded test log parser. Besides that, are there any optimizations that you found particularly helpful?

[njukenanli]

Hmmm... We are thinking about efficient methods that accelerates build and test when number of repos to be processed becomes large...

1. Turning language-specific build and test prompt into skills updatable by the agent at the end of successful exploration.
2. Reuse the built images and extracted test commands/parsers for other PRs of the same repo.
3. Collecting successful trajectories of RepoLaunch for rejection-sampling based fine-tuning of LLM.

We are considering expanding this paper with these potential improvements. But it depends on our free time and computing resources hahaha...

However, these methods will work ONLY when you are batch-processing a large number of repos/PRs. Hope this will give you some insight.