Through developing the auth updates from Tiled I found development was slowed down significantly by the long runtimes of the unit tests. Usual runtime right now is 22 minutes. I managed to get this down to around 10 minutes on my local machine by creating a parallelization system. By I, I mean of course AI. This works by doing the following
- break down unit tests into groups
- spin up multiple test runners, each with the resources to perform all tests, such as queueserver and redis
- begin passing chunks of tests to runners
However even with this strategy the CPU is idle for the majority of the test time, as certain tests take a long time to run and so the runner that is doing those long tests can't really be sped up.
I think there are some strategies we can take to improve run times significantly.
- Break tests down into integration and unit tests. Right now all are run as the same thing. This way developers can run unit tests locally and then run integration tests in CI when they want
- use testcontainers as from what I have read they can spin up dependencies like queueserver and redis as needed. Which could potentially then allow better parallization
- Use parallelization of tests. Tiled does this with xdist
- Remove all time.sleep calls in integration tests. This is already called out as todo's in the unit tests as is
- Increase the number of unit tests to improve coverage so that the non-integration tests are more likely to catch anything the integration tests are verifying
- Focus on making the tests run with just pytest, not having to spin up additional systems/services
- Remove my hacky local parallel runner
In my opinion, unit tests should run within 5 minutes. Integration tests can take longer but really should be faster than they are right now.
Through developing the auth updates from Tiled I found development was slowed down significantly by the long runtimes of the unit tests. Usual runtime right now is 22 minutes. I managed to get this down to around 10 minutes on my local machine by creating a parallelization system. By I, I mean of course AI. This works by doing the following
However even with this strategy the CPU is idle for the majority of the test time, as certain tests take a long time to run and so the runner that is doing those long tests can't really be sped up.
I think there are some strategies we can take to improve run times significantly.
In my opinion, unit tests should run within 5 minutes. Integration tests can take longer but really should be faster than they are right now.