[engine] Eager execution and fully declarative dependencies

Review Request #3339 - Created Jan. 15, 2016 and submitted

Stu Hood
2782, 2825
benjyw, ity, jsirois, nhoward_tw, patricklaw

This is a 96% working refactoring of the experimental engine to:
1. mix planning with execution
2. declare all dependencies of tasks ahead of time

  • Planners are now ("output product type", "input product selection clause", "task function") triples. The selection clause is an ANDed series of Selector instances which declare the dependencies of the function. See the bottom of planners.py for the updated examples.
  • Don't create an execution graph before beginning to execute it: instead, grow a ProductGraph while executing it.
    • While being more difficult to experiment with without causing sideeffects, this model allows for significant power: for one thing, it allows dependency lists to be created/determined by task code. This opens the door to tasks that do dependency inference by parsing input files (up to and including implementing BUILD file parsing inside the engine!).
  • A ProductGraph is made up of nodes which each uniquely identify a computation.
    • The primary reason for the complexity of the Node.step API is that all of the dependencies of some nodes cannot be computed until some of their dependencies become available. In particular a SelectDependencies Node begins by requesting the product it needs to determine the dependency list; once that is available, it requests those dependencies as well.
  • Rename config_selectors to variants, give them names, and make them transitive through the graph (also possible to specify in an address on the CLI)
    • One of the goals of this change is to allow for using the same system to solve three problems:
      1. thrift cross-compilation (as John already demonstrated)
      2. scala/javascript cross compilation (easy: left to the imagination)
      3. frozen ivy resolutions (by configuring a variant on binary targets that travels all the way down to their 3rdparty dependencies)

The visualizer tool continues to work, but now requires that the graph be executed before it can be displayed. Attached is the graph for:

./pants run src/python/pants/engine/exp/examples:viz -- \
  tests/python/pants_test/engine/exp/examples/scheduler_inputs \
  compile \



Nick Howard (Twitter)
Stu Hood
Stu Hood
Stu Hood
Stu Hood
Stu Hood
Stu Hood
Review request changed

Status: Closed (submitted)

Change Summary:

Merged --tbr as ef3f8d221a5afefb01d655448ce7e3f537399810