The Aikau unit tests have grown to the point where it is not always guaranteed that all tests will pass in a single test run (although those same tests will pass in isolation). This issue has been raised to tackle the problem and should not be resolved until all unit tests pass reliably both locally (on Windows, Mac and Linux) and on the Vagrant VM.
Here are some potential ideas that could address the stability problems:
- Stop using the unit test bootstrap page and convert all the JSON test models into WebScripts. Then update the tests so that they load the WebScript directly instead of the unit test bootstrap. Even if this doesn't improve stability, it should significantly speed up testing time. This should be done in conjunction with moving all the tests to either a separate project or to the "test" section of the Maven project
- Break the tests up. Frequently (especially when running on Vagrant) the tests timeout because they only have a certain amount of time to run. It's possible to run multiple tests without loading a new page. By breaking the tests up it means that its easier to identify exactly where the failures are and to avoid the timeout issues
- Improve the tests themselves. Frequently the tests fail to match CSS selectors that are known to exist. The problem could be a result of variable sleeps (it would be better to avoid sleeps entirely) or because the CSS search completes before the classes have been applied. We need to identify the commonly failing tests to understand why they fail