Add support for python test sharding.

Review Request #2243 - Created May 19, 2015 and submitted

Information
John Sirois
pants
jsirois/pytest_run/shard
1569
be8c828...
Reviewers
pants-reviews
benjyw, jinfeng, zundel
The option follows the format of the similar option in the junit-runner
and the sharding happens at the finest-grained test function/method
level.

Use this new feature to replace the target-level sharding implemented
in the ci.sh script.

 .travis.osx.yml                                                  |  5 ++--
 .travis.yml                                                      | 15 ++++++-----
 build-support/bin/ci.sh                                          | 33 +++++++++++------------
 src/python/pants/backend/python/tasks/pytest_run.py              | 71 +++++++++++++++++++++++++++++++++++++++++++++-----
 tests/python/pants_test/backend/python/tasks/python_task_test.py | 18 ++++++++++++-
 tests/python/pants_test/backend/python/tasks/test_pytest_run.py  | 37 ++++++++++++++++++++++----
 6 files changed, 141 insertions(+), 38 deletions(-)

Locally verified the script changes with:

$ CXX=g++ ./build-support/bin/ci.sh -u 0/10 -i 0/10

Also did a look and feel on the console for sharded vs non output.

Non-sharded:

$ pants.dev test.pytest --shard=0/1 tests/python/pants_test/backend/python/tasks:
...
02:40:29 00:00     [pytest]
02:40:29 00:00       [run]
                     ============== test session starts ===============
                     platform linux2 -- Python 2.7.8 -- py-1.4.27 -- pytest-2.6.4
                     plugins: timeout, cov
                     collected 46 items 

                     tests/python/pants_test/backend/python/tasks/test_setup_py.py .............
                     tests/python/pants_test/backend/python/tasks/test_pytest_run.py .............
                     tests/python/pants_test/backend/python/tasks/test_python_repl.py .......
                     tests/python/pants_test/backend/python/tasks/test_python_eval.py .............

                     ===== 46 passed, 1 warnings in 49.38 seconds =====

Sharded - note extra test session header lines explaining the shard:

$ pants.dev test.pytest --shard=0/2 tests/python/pants_test/backend/python/tasks:
...
02:38:08 00:00     [pytest]
02:38:08 00:00       [run]
                     ============== test session starts ===============
                     platform linux2 -- Python 2.7.8 -- py-1.4.27 -- pytest-2.6.4
                     shard: 0 of 2 (0-based shard numbering)
                     plugins: timeout, cov
                     collected 46 items 
                     Only executing 23 of 46 total tests in shard 0 of 2

                     tests/python/pants_test/backend/python/tasks/test_setup_py.py .......
                     tests/python/pants_test/backend/python/tasks/test_pytest_run.py ......
                     tests/python/pants_test/backend/python/tasks/test_python_repl.py ....
                     tests/python/pants_test/backend/python/tasks/test_python_eval.py ......

                     ===== 23 passed, 1 warnings in 39.47 seconds =====

$ pants.dev test.pytest --shard=1/2 tests/python/pants_test/backend/python/tasks:
...
02:39:13 00:03     [pytest]
02:39:13 00:03       [run]
                     ============== test session starts ===============
                     platform linux2 -- Python 2.7.8 -- py-1.4.27 -- pytest-2.6.4
                     shard: 1 of 2 (0-based shard numbering)
                     plugins: timeout, cov
                     collected 46 items 
                     Only executing 23 of 46 total tests in shard 1 of 2

                     tests/python/pants_test/backend/python/tasks/test_setup_py.py ......
                     tests/python/pants_test/backend/python/tasks/test_pytest_run.py .......
                     tests/python/pants_test/backend/python/tasks/test_python_repl.py ...
                     tests/python/pants_test/backend/python/tasks/test_python_eval.py .......

                     ===== 23 passed, 1 warnings in 25.95 seconds =====

And a manual OSX CI launched with:

$ cp .travis.osx.yml .travis.yml && \
  git commit -am 'Manual OSX CI.' && \
  git push -f git@github.com:pantsbuild/pants-for-travis-osx-ci.git HEAD:master && \
  git reset --hard HEAD~1

...went green here:
https://travis-ci.org/pantsbuild/pants-for-travis-osx-ci/builds/63141484

Finally, linux CI went green here:
https://travis-ci.org/pantsbuild/pants/builds/63232758

Issues

  • 0
  • 0
  • 1
  • 1
Description From Last Updated
John Sirois
Eric Ayers
John Sirois
Eric Ayers
John Sirois
John Sirois
John Sirois
Review request changed

Status: Closed (submitted)

Benjy Weinberger

   

Sorry for the late reply - we're on an offsite right now. But I have an alternative solution for this in a branch that I had been working on late last week (I know, you snooze you lose...)

In my solution each test has its own workdir, but they all share an interpreter/pex/artifact cache dir. This is achieved by making the cache dir (previously "scratch dir") location an option, and setting it to point to the real pants instance's cache dir.

It seems to work really well, with the same or better speed savings.

Does this sound like a better solution? I can think of a few advantages: A) It's less hacky, B) each test still gets a pristine workdir. C) When iterating on the same test(s) over and over in a debugging session you don't incur the cost of even a single interpreter setup. D) I haven't done this yet, but presumably even integration test pants subprocesses could be run against this shared cache dir, and they'd get a lot faster too.

OTOH I'm not sure what the concurrency implications are with multiple test shards using the same cache at the same time. But we could deal with that somehow.

Thoughts?

  1. Sorry, didn't mean to open an issue for this, just have a convo.

  2. I do like the idea of each test having its own workdir, and have to admit I was concerned about this portion of it until I made sure that the logic doesn't apply to the tasks I've spent the most time working on which are not covered by this superclass.

  3. Sounds good to me.
Loading...