Parameterize the test runner so it can re-run all tests with paralleism enabled.

Review Request #3920 — Created May 21, 2016 and submitted

benjyw, gmalmquist, jsirois, stuhood

This is a bit more infrastructure around testing before landing the experimental test runner.
- Parameterizes the ConsoleRunnerTestBase so it can re-run all tests with parallelism enabled by default.
- Moves some tests around, making a separate XMLReport test.
- Moves some test classes from the impl to the lib package.
- Remove the -parallel-methods argument. It was marked experimental
- Prepares for re-running all tests with the experimental test runner on.

Tests green at

  • 0
  • 0
  • 1
  • 0
  • 1
Description From Last Updated
  2. Locally, I've found these shard tests to be flaky when run 6 times in one JUnit run. All of the tests get run, but in different shards. Haven't gotten to the bottom of that yet.

  2. tests/java/org/pantsbuild/tools/junit/impl/BUILD (Diff revision 1)

    Should prefer 1:1:1 and move this file out to its own package/directory; not a strong objection though.

    1. Actually I had tried this before and reverted it because the Xml tests required so many package private methods. But then I refactored the Xml tests. In the end this turned out to be pretty easy, I just had to open up the visibility on two methods.

  2. Why the switch to LinkedHashMap? AFAICT we never iterate over the entries, so iteration order shouldn't matter.

    Also, can the variable type be a Map instead of a HashMap?

    1. I reverted this change and made the variable a Map.

      I've been having some issues in this area of the code (sharded tests are returning different tests in different shards) and am still trying to get to the bottom of it.

    2. Ah Ha! I think I just found the source of non-determinism in Class.getMethods() used down in legacyParseRequests(). This is some new code I added recently to support the PARALLEL_METHODS mechanism of running tests so it shouldn't impact anyone else.

  1. Ship It!
Review request changed

Status: Closed (submitted)

Change Summary:

Thanks Stu and Benjy. Commit 76c8411