[engine] Optionally inline inlineable Nodes

Review Request #3931 - Created May 25, 2016 and submitted

Stu Hood
jsirois, kwlzn, nhoward_tw, wisechengyi

This review adds support for inlining execution of Nodes, which skips storing them in the ProductGraph between execution, or involving them in the scheduling loop. The attached viz shows that only TaskNode and FilesystemNode (and a root DependenciesNode) are still stored in the ProductGraph... other nodes have been inlined.

It has fairly nice implications for performance... with caching disabled, cold ./pants list :: with the daemon is down to ~8 seconds on my machine, which puts it at around 5x of cold without the daemon. It also makes the remaining hotspots in the profiles significantly more obvious.

  • Remove the step(.., dependencies) argument by hiding it behind StepContext.get.
  • Add Node.is_inlineable, to indicate that a Node runs cheaply enough to not be worth memoizing in the ProductGraph.
  • Optionally inline inlineable dependencies in StepContext.
  • Move the Selector->Node factory to StepContext, which eliminates some unnecesary cross-module imports.
  • Switch legacy.graph to consuming the addresses of the root LegacyTarget nodes, and back to maybe_list (yep... sorry Yi).
  • Add ProductGraph.trace, and use it to log stacktraces.
  • Lazily format Noop messages, since they are rarely used.
  • Render cyclic/rejected dependencies as dotted lines in ProductGraph.visualize.



Kris Wilson
John Sirois
Stu Hood
Stu Hood
Review request changed

Status: Closed (submitted)

Change Summary:

Merged as 5cf0104a73756d8b5e3fa33a77cf3d64926976eb