Simplify and robustify `LocalMultiprocessEngine`.

Review Request #3084 - Created Nov. 5, 2015 and submitted

Information
John Sirois
pants
jsirois/issues/2496
2496, 2507
Reviewers
pants-reviews
kwlzn, nhoward_tw, stuhood
This change removes the use of a thread and queues above multiprocessing
and switches to polling of async results for 2 benefits:
1. Both pool submissions and result retrievals are perfomed from the
   main thread and, as such, low-level multiprocessing errors - namely
   pickling errors - are naturally raised in the forground making new
   planner development easier to debug.
2. The re-shaped forground pool interaction better saturates the pool
   and it does this via a more directly obvious algorithm.

 src/python/pants/engine/exp/engine.py             | 168 ++++++++++++++++++++++++++++----------------------
 src/python/pants/engine/exp/examples/planners.py  |  41 +++++++++++-
 tests/python/pants_test/engine/exp/test_engine.py |  28 +++++++++
 3 files changed, 164 insertions(+), 73 deletions(-)
CI went green here:
  https://travis-ci.org/pantsbuild/pants/builds/89710858

Issues

  • 0
  • 1
  • 0
  • 1
Description From Last Updated
John Sirois
Stu Hood
John Sirois
John Sirois
Stu Hood
John Sirois
John Sirois
John Sirois
John Sirois
Review request changed

Status: Closed (submitted)

Loading...