Add a visualization tool for execution plans.

Review Request #3010 - Created Oct. 22, 2015 and submitted

John Sirois
918, 2421
benjyw, ity, kwlzn, nhoward_tw, patricklaw, stuhood
This adds some APIs and a simple tool for visualizing execution plans.
The example planners are moved to `src/python/pants/engine/exp/examples`
and imports and BUILD deps adjusted.

 src/python/pants/engine/exp/examples/BUILD                                                                 |  37 +++++++++++++++
 src/python/pants/engine/exp/examples/                                                           |   0
 tests/python/pants_test/engine/exp/ => src/python/pants/engine/exp/examples/ |   0
 src/python/pants/engine/exp/examples/                                                         | 124 ++++++++++++++++++++++++++++++++++++++++++++++++++
 tests/python/pants_test/engine/exp/BUILD                                                                   |  19 +-------
 tests/python/pants_test/engine/exp/                                                          |   2 +-
 tests/python/pants_test/engine/exp/                                                       |   4 +-
 7 files changed, 166 insertions(+), 20 deletions(-)

Locally used the following - graph attached:

$ ./pants run src/python/pants/engine/exp/examples:viz -- \
  tests/python/pants_test/engine/exp/examples/scheduler_inputs compile src/java/codegen/simple

CI went green here:


John Sirois
John Sirois
John Sirois
John Sirois
John Sirois
Review request changed

Status: Closed (submitted)

Stu Hood

Handy. I think I'm able to see now how the combination of products and subjects allows for targets to be located at multiple points in thr graph (to be both the compiler, and the compiled). Exciting!

Are you using the 'subject' name to differentiate from target in some way (to support a superset of targets, perhaps?) or is it just to avoid confusion?

  1. Subject is where synthetic targets went.  Typically a Subject just points to a primary and that is a target, but need not be.  The Jars you see in the IvyResolve plan box are in-line Jar objects w/o an address for example.  Those don't come from any real target, but from the ApacheThriftConfiguration for java.  The ApacheThriftPlanner injects those dependencies for others to resolve by returning a plan whose subject adds an alternate to the primary subject of the requested promise for Sources.of('.java') by the JavacPlanner:
    This was the hardest part for me to fogure out - how to make it so that ApacheThriftPlanner did not need to know about ivy or setuptools or any other resolver system.  With the ability to propose an alternate (target) it only needs to know about lists of deps, not even what their type is or what they mean, just that codegen for lang X depends on these blobs.

Is this because we're going to continue to support untyped products, keyed by a string?

  1. No - it was just to deal with a temporary hack fixed here:
    `Sources.of(ext)` is now back as a product_type factory since I've learned me more about pickling (needed for multiprocess engine).